Episode Description

The relationship between AI and ethics is both developing and delicate. On one hand, the GenAI advancements to date are impressive. On the other, extreme care needs to be taken as this tech continues to quickly become more commonplace in our lives. In today’s episode, Ovetta Sampson and I examine the crossroads ahead for designing AI and GenAI user experiences.

While professionals and the general public are eager to embrace new products, recent breakthroughs, etc.; we still need to have some guard rails in place. If we don’t, data can easily get mishandled, and people could get hurt. Ovetta possesses firsthand experience working on these issues as they sprout up. We look at who should be on a team designing an AI UX, exploring the risks associated with GenAI, ethics, and  what people designing these applications need to be thinking about going forward.

Highlights/ Skip to:

• (1:48) Ovetta's background and what she brings to Google’s Core ML group
• (6:03) How Ovetta and her team work with data scientists and engineers deep in the stack
• (9:09)  How AI is changing the front-end of applications
• (12:46) The type of people you should seek out to design your AI and LLM UXs
• (16:15) Explaining why we’re only at the very start of major GenAI breakthroughs
• (22:34) How GenAI tools will alter the roles and responsibilities of designers, developers, and product teams
• (31:11) The potential harms of carelessly deploying GenAI technology
• (42:09) Defining acceptable levels of risk when using GenAI in real-world applications
• (53:16) Closing thoughts from Ovetta and where you can find her

Quotes from Today’s Episode

• “If artificial intelligence is just another technology, why would we build entire policies and frameworks around it? The reason why we do that is because we realize there are some real thorny ethical issues [surrounding AI]. Who owns that data? Where does it come from? Data is created by people, and all people create data. That’s why companies have strong legal, compliance, and regulatory policies around [AI], how it’s built, and how it engages with people. Think about having a toddler and then training the toddler on everything in the Library of Congress and on the internet. Do you release that toddler into the world without guardrails? Probably not.” - Ovetta Sampson (10:03)
• “[When building a team] you should look for a diverse thinker who focuses on the limitations of this technology- not its capability. You need someone who understands that the end destination of that technology is an engagement with a human being.  You need somebody who understands how they engage with machines and digital products. You need that person to be passionate about testing various ways that relationships can evolve. When we go from execution on code to machine learning, we make a shift from [human] agency to a shared-agency relationship. The user and machine both have decision-making power. That’s the paradigm shift that [designers] need to understand. You want somebody who can keep that duality in their head as they’re testing product design.” - Ovetta Sampson (13:45)
• “We’re in for a huge taxonomy change. There are words that mean very specific definitions today. Software engineer. Designer. Technically skilled. Digital. Art. Craft. AI is changing all that. It’s changing what it means to be a software engineer. Machine learning used to be the purview of data scientists only, but with GenAI, all of that is baked in to Gemini. So, now you start at a checkpoint, and you’re like, all right, let’s go make an API, right? So, the skills, the understanding, the knowledge, the taxonomy even, how we talk about these things, how do we talk about the machine who speaks to us talks to us, who could create a podcast out of just voice memos?” - Ovetta Sampson (24:16)
• “We have to be very intentional [when building AI tools], and that’s the kind of folks you want on teams. [Designers] have to go and play scary scenarios. We have to do that. No designer wants to be “Negative Nancy,” but this technology has huge potential to harm. It has harmed. If we don’t have the skill sets to recognize, document, and minimize harm, that needs to be part of our skill set.  If we’re not looking out for the humans, then who actually is?” - Ovetta Sampson (32:10)
• “[Research shows] things happen to our brain when we’re exposed to artificial intelligence… there are real human engagement risks that are an opportunity for design.  When you’re designing a self-driving car, you can’t just let the person go to sleep unless the car is fully [automated] and every other car on the road is self-driving. If there are humans behind the wheel, you need to have a feedback loop system—something that’s going to happen [in case] the algorithm is wrong. If you don’t have that designed, there’s going to be a large human engagement risk that a car is going to run over somebody who’s [for example] pushing a bike up a hill[...] Why? The car could not calculate the right speed and pace of a person pushing their bike. It had the speed and pace of a person walking, the speed and pace of a person on a bike, but not the two together. Algorithms will be wrong, right?” - Ovetta Sampson (39:42)
• “Model goodness used to be the purview of companies and the data scientists. Think about the first search engines. Their model goodness was [about] 77%. That’s good, right? And then people started seeing photos of apes when [they] typed in ‘black people.’ Companies have to get used to going to their customers in a wide spectrum and asking them when they’re [models or apps are] right and wrong.  They can’t take on that burden themselves anymore. Having ethically sourced data input and variables is hard work. If you’re going to use this technology, you need to put into place the governance that needs to be there.” - Ovetta Sampson (44:08)

Links Referenced

• Black Design Leadership Program: https://blackdesignleadership.com
• Ovetta’s LinkedIn: https://www.linkedin.com/in/ovettasampson/
• Episode 42: https://designingforanalytics.com/resources/episodes/042-why-machine-learning-and-analytics-alone-cant-drive-behavioral-change-inside-police-departments-with-allison-weil/ 
• Episode 70: https://designingforanalytics.com/resources/episodes/070-fighting-fire-with-ml-the-ai-incident-database-and-why-design-matters-in-ai-driven-software-with-sean-mcgregor/ 

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, I am happy to say I finally have Ovetta Sampson on the line, Director of UX for Core Machine Learning at Google. Ovetta, I have been wanting to have you on the show for a long time, so I’m really happy that you’re here. Thanks for making time.

Ovetta: Well, I have to say, I want to thank you for your persistence. It’s been kind of crazy. My APB, Jamie, shout-out. I always have to give her a shout-out on these because she works miracles with time. And I know we haven’t, like, achieved time travel yet, but she’s really close. Like—

Brian: [laugh].

Ovetta: —she really understands how to work magic with time, and calendars, and stuff like that. My calendar is a hot mess, and so she somehow finds all the time to fit in all the things that I’ve been asked to do. So, I’m really excited to be here, finally, on the podcast, and hopefully it will be worth it for your audience.

Brian: Yeah, yeah, yeah. I’m sure it will be. If anything, if it’s worth it to me, then, hey, you’re getting this for free, listeners, so—[laugh]

Ovetta: [Well 00:01:46].

Brian: But anyhow, this is going to be worth more than free. I mean, you’ve been a swim coach, a reporter, you’re the founder of the Black Design Leadership Program. You were VP of ML and Responsible AI at Capital One. Now, you’re leading a UX team at Google on the Core Machine Learning, and you’re teaching at DePaul on human-centered AI design. I love the diversity here. Why do you care about designing for AI, and why does Google care that they need to have user experience involved with core machine learning for people that are building machine learning applications and solutions? Why do you exist there?

Ovetta: [laugh]. Yeah, it’s funny. I was on a panel at CES last year, and it was on developer experience, and I remember saying this—it was an audience of developers—and it was just about, kind of like, software development, and how developer experience is, kind of like, created, and how do we create tools for developers. And I remember saying that developers are people too, and the whole room just, kind of like, erupted in laughter. But I think it’s really funny because we forget that people who create all of these amazing digital tools, and software, and AI, and ML, and all of these amazing tech fuelled tools are people.

And when there’s people in technology, there’s the right formula for user experience. And so, it doesn’t matter that my team is really deep in the stack. Like, my team deals with user experience challenges from everything from how we create and compute chips and data centers all the way through how we put machine learning models into production. So, that’s a wide range of different challenges and problems. And when you have humans engaging with technology along that spectrum, you have the opportunity for amazing user experience.

Brian: I love the framing. I talk about this as well. I don’t even use UX a ton because it almost feels like developers and technical people or analysts are not users. Those are people that are further [laugh] downstream. And I like the human-centered wording better because it’s inclusive of anyone touching the solution. In your case, you’re building the tools to build the solutions, and that counts, as well. What’s good design mean in that context? Like, what is good design for core machine learning? What’s bad design for core machine learning? Help our listeners understand that?

Ovetta: Yeah, I think when we’re talking deep in the stack, and when we’re talking about the meta part of technology where we’re building tools for the builders who build the technology for other people that people access, we really hit on what I think is a core of human-centered design, and that is, amplification and enablement. So, really good user experience amplifies human ability, and enables humans to do more than they would on their own. So, it’s technology using that, and designing that, and shaping that so that a software developer can 10x his or her effect on the world. I’ve been in enterprise tool design all of my career. It’s not the sexy one. It’s not the one that people talk about because most people talk about consumer-facing design.

Which is awesome, like, Apple has a high reputation for that. I worked at IDEO. IDEO has a high reputation for that, and consumer-facing, kind of, design is amazing. But I just really like designing tools for the builders, for the crafters, for the creators. It’s a fun thing to do. It’s a fun group of user groups, very diverse.

And when we get deep in the stack, it’s very technical, but at the end of the day, it’s just humans and tools and amplification and enablement. I’m really proud to say that the tools that we’re working on are used by people at Google DeepMind to create AlphaFold, right, which just won the Nobel Prize for Chemistry. So, when we talk about deep in the stack infrastructure tooling, we’re talking about enablement and amplification of some amazing creators.

Brian: Could you talk a little bit about how your team gets access to these people? Do you have enough, just at Google alone? I would imagine you probably tap some of your own, do some dog-fooding, but how do you get in touch with the data scientists, the ML engineers? How does your team work with them?

Ovetta: Yeah, it’s funny because I like to say that I have the smartest user group in the world, right? 90% of the things that we design end up in the hands of a Google Developer, a Google employee, a designer, technical lead, TPM, data scientist, data analyst. And so, our user group is pretty rarefied, always on the cutting edge, but also, like, it can be kind of a bubble, too. Google has very unique challenges and complex challenges, so the tools that we build are always for scale. And so, sometimes when you’re building for scale, you can forget about scrappy prototyping.

And so, I have to remind our team to get lean, and scrappy, and do multi-phasing concept and validation launches, instead of saying it has to be perfect and bulletproof, you know, to scale. And so, the researchers on the team that I lead are amazing. They are very connected to our different types of users. We spend an enormous amount of time observing them, interviewing them, working with them to co-design some of the tools and services and platforms that we are responsible for at Google. And our philosophy is very human-centered.

We realize that our user group is really smart, that they can build their own tools—and they do, often—but we look at them and we say, what is it that you really want to spend your time on, that you really want to pour your craft into, and let us take care of everything else that enables you to get the job done. And so, that’s kind of how we work with our user group on trying to discern what is important to them, what is important to the business at Google, what is important to what’s good for Google is also kind of good for open-source and other folks, so we definitely have our eye on that. And it is a pleasure to work at a company that when you do build something, that it can end up with millions of users.

Brian: Yeah, yeah. I like the frame what you talked about. It’s letting people focus on—you know, Jared Spool uses this language of ‘goal time’ and ‘tool time’—

Ovetta: Yeah.

Brian: —trying to let the software handle as much of the tool part [laugh], so you can focus on the goal time, and the work you really want to do. If you’re a data scientist, it’s probably modeling and not data engineering, and data cleansing, and data ingesting, and ETL, and running SQL, and all that kind of stuff. Do you have a thought, too, around how we design for AI on the recipient side, so on the user side? Do you have any thoughts about how you perceive that as being a different craft or a different approach than with non-probabilistic interfaces? So, this is—now we’re, kind of, shifting a little bit more to the end-facing application side of things. How do you think about that today? Is it different, is it really the same? Like, new technology, same guiding principles of design still apply. Or, like, no, no, no, no, no, like, the game has—the rules are now different. You know, it’s not baseball, it’s softball. You know, it’s—I don’t know.

Ovetta: I had to take my two-year Google conduct training, whatever. It’s like, how to be a good Googler and all that good stuff. One of the things that’s in that conduct training is understanding of our AI policies. And so, if artificial intelligence was just, kind of, another technology, why would we build entire policies and frameworks around it? The reason why we do that is because we realize that there are some real thorny ethical issues that are wrapped into this amazing capability that we call artificial intelligence.

And first and foremost is data. Who owns that data? Where does it come from? And I like to say that data is the love language of machines, right? And so, machines could not learn without data, and then they can’t solve problems without data, but data is created by people, and all people create data. And so, there’s this very, kind of like, when you’re talking about the end-user, you’re also talking about your end co-designer, right? The data that is created by all the people that go into large language models and all that good stuff is part of the design process of machine learning.

And so, I think it’s really important that we understand that this technology is, more than any other technology, takes a lot of us to make it work. And when I say, “Us,” I mean our data, our behavior, but also our craft and our creation, right? Whereas, like, software—which is non-dynamic, it’s pretty stoic, it does what you want it to do, right, and it executes until you change it, right—AI learns from us, and then it executes, right? It makes decisions. And so, it’s a little bit different. It’s not like your father’s technology, right?

And that’s why companies have strong legal, compliance, regulatory, as well as policies around the use of this technology, how this technology is built, and then how we want to design it to engage with people. Because I like to say, think about having a toddler, and then train the toddler on everything in the Library of Congress, plus the internet. Like, do you just want to release that toddler into [laugh] the world without having some guardrails? Probably not, right? And so, that’s the opportunity for design.

When I was at IDEO, we created AI ethical principles, design principles in 2016, and one of them was—one of our four laws was, unintended consequences are opportunities for design when it comes to AI. Because we can’t predict everything that’s going to happen, but when it does, that’s where design really needs to show up. How can we minimize any human engagement risks that come with this technology, and we do that through design.

Brian: Yeah. Let me throw a scenario at you. Let’s say you have a former colleague from Capital One. They’re at a new place now, they’re in product management, or maybe they’re at a startup or whatever, and they’re building some kind of a product that leverages AI heavily, so information, it could be generative, it may not. It could be predictive analytics, something like, I don’t, it doesn’t really matter what flavor of it is. Maybe they don’t even know yet.

And they’re asking you over coffee, like, “Hey, you know, my developers are kind of been doing all the design here, and you know, I don’t feel like our vision is coming together here. Like, what kind of designer or agency or skill set should I be looking for for this kind of thing? Because our whole product is really driven by artificial intelligence. That’s the magic sauce that puts us—that’s how we’re different than the market because we can do X, Y and Z. What do I need? What should I be looking for in this kind of person, agency, employee? What should I look for?”

Ovetta: Yeah, you should look for a diverse thinker who focuses on the limitations of this technology and not its capability. And that’s a paradigm shift because a lot of folks who build models—and I count myself as one who plays one on TV. You know—

Brian: [laugh].

Ovetta: —I’m not a doctor, but I play one on TV. Like, I can build a model, but, you know, it’s not… it’s not my thing. But the folks who do are really steep and deep into the technology itself and how it can be crafted and created to solve problems. And within that team, you need someone who understands that the end destination of that technology is an engagement with a human being. And you need somebody who understands how human beings engage with machines and digital products, right? And you need that person to be passionate about testing various ways that relationship can evolve.

Because when we’re talking about when we go from software to decision-ware, right, when we go from execution on code to machine learning, we make a shift from agency being powered by the person to a shared agency relationship, meaning the user and the machine both have decision-making power. Now, the decision-making power is limited by the machine—by how we design it—but it’s machine learning, so it can learn, even if we didn’t design it that way, right? And that’s the paradigm shift that people really need to understand. And when that happens, the machine has limitations that the human that it’s engaging with may not understand, and that’s the opportunity for design. So, you want somebody who really understands that gulf, but also that connection between the human and the machine, and can keep that duality in their head as they’re testing product design.

Brian: Yeah. You said earlier, like, data is the love sauce of—

Ovetta: Yeah, data is the love language of machines.

Brian: Yes. So—

Ovetta: Yeah. They don’t run without data, right?

Brian: Yeah. My general sense, I have half a foot in the design and UX community these days. Most of the time I’m kind of hanging in the data and the product world, but when I dip back into that world, I generally feel like the conversation around designing for machine learning and analytics, and for artificial intelligence, that conversation is 90% about using this tooling to do design work, and there’s very little conversation about designing products that use these solutions. There’s very little conversation about data as a strategic driver for the business. Like, how could we leverage all this stuff that we have to increase customer value or to produce new benefits in the world, whatever it may be. And then it comes down from the top, which is what’s our GenAI strategy?

Ovetta: Yeah, yeah, yeah.

Brian: Plop. You know? And the swoop and poop of the AI plop comes in. And then it’s scramble, scramble, scramble, put a prompt box somewhere into the product where someone types stuff in, and that’s the beginning of AI design at fill-in-the-blank company.

Ovetta: Right.

Brian: Are you seeing this as well? I feel like design is really behind on being aware of this, and now it’s just, like, fast catch up. It’s almost like chasing, like, 20 years ago, when we’re chasing engineer—you’re trying to get ahead of engineering. Now, it’s like you’re trying to get ahead of this thing. And most of the conversation is still about, is Figma going to—is that going to replace my job? It’s all about the tooling [laugh].

Ovetta: Yeah, I think, you know, we’re at that stage though. And I remember saying this, I was on a panel yesterday, and I said, you know, we’re at rock, fire stage. Like—

Brian: [laugh].

Ovetta: —we’re Neanderthals walking around, like, [clicking 00:17:59] rocks together, and we’re like, “Whoa, there’s a spark.” Five, ten years from now, though, when the craft catches up with the technology, it’s going to be some transformative stuff. And I always say that AI’s value will be fully realized when it’s connected to other powerful technology like quantum computing, AR/VR, when people really start making those connections, it’s going to be a superpower kind of booster for a lot of things that we turn to technology for. And that could be big problems, like climate change, clean water, stable housing. Like, if you think of all the big stuff, once you add AI to some other, kind of like, stimulus, that big stuff can be tackled, I think.

I think where we are right now is, we’re at that stage. I like to think about artificial intelligence and its growth, compare it to medical and biomedical technology. If you think about, like, the first test tube baby was in 1978, but that wasn’t the first time we were testing out replicating DNA and biology. Like, we were doing it for centuries, right? For years. And so, it wasn’t until the first test tube baby came out that there was this explosion in medical and biomedicine that ended up in CRISPR right, like, now where, actually, really, a machine is creating DNA and all that good stuff, and you have cloning and blah, blah, blah.

But it was like slow moving down the river, and then you had this amazing blip of biomedicine where a computer scientist and a doctor got together to give an English couple their dream, right? And now you’re seeing that again with AI getting together with chemistry and recognizing proteins, and all that good stuff. So, I think we’re at the moment where we’re doing the velociraptor thing, where we’re just wrapping generative AI around whatever use case we think is good to try to find the sweet spot for this technology. And it’s a fun time to be in design because now we can reimagine all the challenges that we had with this superpower and figure out how we can solve them. And one of the things that I think, as a designer, when it comes to AI that is really important for us to remember is, the end result should augment or amplify humanity.

We can’t let the technology outrun that purpose of design, at least in my philosophy. I think if we do, if we put the tech first, we’ll keep sitting in this rock, fire, velociraptor, kind of like, application phase, and not really doing anything to benefit society. I love Gemini. I think it’s awesome, but it’s not meant just to rewrite my resume. There’s going to be a breakthrough there that I often talk about, a colleague of mine who I used to work with at Microsoft who didn’t know much about artificial intelligence, but was really smart in hardware, and he made an AI nose, basically. He made a device using Raspberry Pi, an algorithm, a classification of chemicals, and oxygen that could smell, like, coffee, and whiskey, and all that good stuff.

And I thought that was great. And he wrote about it in a maker’s magazine, and this 14, 15-year-old kid in LA saw that, and he took it, and he took the same kind of maker recipe, and he made a pneumonia sniffer using AI that could detect people who were sick by the smell of certain gasses and stuff like that. And that’s the kind of thing that I feel like we’re going to get more of, where makers, and tinkers, and really smart people, and the one-percenters of what I call, like, the GoogleDeep mines and all those folks are going to make stuff, and then they’re going to show it to folks, and then other people are going to make stuff. You know, grandmas are using Gemini, you know, for recipes, and you know, stuff like that. And then we’re going to keep creating things, and then suddenly we’re going to see our world transformed.

Brian: You know, even thinking back to, like, this fictitious—your product manager friend who’s looking to hire—get some help or whatever, do designers need to understand much about how these models work, how they produce the insights that they do or the predictions that they do, how they classify looking back? And I’m sure you’ve seen and heard plenty of debates about should designers code or not? The old engineer-design thing. And there’s the camp of the designers, like, I shouldn’t need to know about that changes my creativity, and I don’t want to be [hurtled 00:23:06] by what’s possible. I need to stay open.

And then you have the camp that’s just like, how can you possibly design when you don’t know the medium that you’re designing for, which is software engineering? Like, you are a much more efficient when you understand what it’s like to be a developer building this thing, and you’re not asking for ridiculous requirements that are completely unfeasible. Two camps there, right? Regardless of where you are in that, I kind of think about AI, in that we’re kind of having that thing could exist again, where it’s the camp of, like, I don’t really need to know about how these things work. I’ll just experiment and play around.

There’s maybe that camp, and then there’s the no, I need to understand something about how did it come up with these predictions, and like, where’s the data that’s coming into this? And should we be looking at the ingest point? Is that part of the reason why we’re getting—you know, it’s giving weird predictions, or, you know, it’s generating these really weird stuff, and it’s like, oh, there’s a user experience problem at the point of intake where we captured this data. Do you have a feeling about where skills should be? Should they be technical as well as the classic design skill? Do we need both of those or not?

Ovetta: Look—and I said this, I think last year—we’re in for a huge taxonomy change. So, there are words that mean very specific definitions today: software engineer, designer, technically skilled, digital, art, craft, all of these things in our brain. I say those words, and I’m sure that you and I are close on what they mean. AI is changing all that. It’s changing what it means to be a software engineer. Remember we talked about earlier? You know, machine learning used to be the purview of data scientists only, right? Like, they were the ones who did statistical models, and they were the ones who selected the algorithms, and validated them, and trained the data and all that good stuff.

Well, with generative AI, all of that is baked in to Gemini. So, now you start at a checkpoint, and you’re like, all right, let’s go make an API, right? So, the skills, the understanding, the knowledge, the taxonomy even, how we talk about these things, how do we talk about the machine who speaks to us talks to us, who could create a podcast out of just voice memos? So, my mom died in March, and I’ve been really hesitant to—there have been folks who use voicemails to create an AI to create their long-lost loved ones. And I’m not there yet. I don’t want to do that.

But Google has this product called NotebookLM, which I love. I’ve loved it ever since—a shout-out to David [Westrun 00:25:53] and the team, all that—loved it ever since it came out, as a writer. And it’s meant to kind of, like, organize your thoughts and, like, help you write and all that good stuff, something that I think AI is really good at because AI is really—machine learning is really great at recognizing patterns, right? It’s really great at synthesis, it’s all that. And as a researcher, former researcher, synthesis is, like, I’m obsessed with it.

And so, I upload it, and now they added audio over it. And so, what it does is, it takes your—so Notebook, takes your text, your images, your even audio, whatever, and it turns it into this—it synthesizes it in the form of a podcast, two people talking, right? That outcome of the podcast to people talking, what do we call that? Is it a machine? Is it an interactive conversation?

Like, that is the something. Like, it has a brain, it made decisions, it synthesized all of that stuff like a person would or whatever, but it’s not a person. It’s not exactly a machine because it lives on your desktop or whatever, but it gave you an outcome, and it was a really amazing, tender outcome that it took my mom’s voicemail and made this commentary about our relationship, just by listening to how my mom talks about me. She used to always say, “Hey, big girl.” Like, that was her thing, and it picked up on that, right? It was like, this is a tender relationship that these people have.

So, it did all the things that a human would do, but it’s not a human, so what do you call that? And that’s what I mean about, like, taxonomy. Like, again, when I say AI is transformative, we haven’t gotten there yet. It’s going to change how we view, see, think, and talk about our world in very transformative ways. So, when we talk about what are the skill sets that people need to be able to evolve with this technology, the first is the ability to think differently about the world, to paradigm shift.

One of the most difficult things that developers have when it comes to machine learning is the fact that it’s a dynamic technology, that the way that you train it, the outcomes that you get before it goes online and into production could and often are different. That is not true in software.

Brian: Yeah. It shouldn’t be [laugh].

Ovetta: That’s a huge shift, right? It shouldn’t be, right? If it is, it’s a bug, right? That’s a huge shift. What kind of technology—and with designers, you can design a UI, but with AI as a user, I can redesign it even before it goes to scale. I can have a UI that’s so hyper-personalized it looks nothing like you—

Brian: Yeah.

Ovetta: —like yours. Like, these are the things if we’re really going to, kind of like, harness the power of artificial intelligence, it’s going to take huge leaps in our mindsets about how our world can be different. So, when I teach to my students, they have to do an AI product, and the first thing I tell them is, I’m going to take points off if the product ends up on your phone. I don’t take points off. But it’s for them to be, like, you literally can design in air with AR and VR, right?

Like, every surface that we see is a potential landing surface for your application. Our phone is not the only thing that carries our digital self anymore. But again, like, how many applications are not going to have phones on it for the next two or three years? So, for me, the skill set is the ability to think differently about your world and to understand how this technology can make that happen. That’s innovation, right?

When that happens, like, when I say it’s a different world, it’s going to be a really different world. And I remember talking-I was on a panel at CES, and I remember telling Harvey Mason, who’s the CEO—junior—of the Recording Academy, and they said, “What is the future going to be like?” And I’m like, “You’re going to have a category for AI-generated content, music. Generative AI music. Non-real people making music,” right?

If you don’t, it’s happening anyway, right, so how do we embrace that this is a new part of art, music, craft, plays, theater, right? This is a new part of that. People got mad when folks were entering art that was partially generated by Midjourney, right? That’s just going to be a norm in two to three years. Because why wouldn’t we use it?

So now, I’ve stayed away from the nuclear bomb of ethics and morals, and we can get to that, but I’m just speaking practically as a person who understands how this technology works, but more so who understands how humans work [laugh]. And that is the key. It’s not about the AI; it’s really us. And we sometimes have no boundaries, or our boundaries are sifting in the sand of what can I do to get what I want, and how can I amplify what I want using this technology? And so, we have to be really kind of—again, like, we have to look out for that because there are bad actors, right? And not even just bad actors; there’s just, like, people, just apathy is something that could let this technology harm humans, right?

So, we have to be very intentional, what I call mindful AI about when we’re building these tools, and that’s the kind of folks that you want on teams like mine, who are willing to go there and play scary scenarios. Because we have to do that. That’s part of design, right? No, designer wants to be negative Nancy, right, but this technology has huge potential to harm, and it has harmed, so if we don’t have the skill sets to, A, recognize that, B, document that, and C, minimize that, that needs to be part of our skill set. Because who’s going to do it? If the designer doesn’t do it, and it’s our job to shape and craft and design tools for humans, if we’re not looking out for the humans, then who actually is?

Brian: Yeah. I mean, in my audience is mostly B2B, enterprise, that kind of thing, and most of the time, designers are the only ones talking about this. When we talk about harm, we’re usually talking about, is there financial risk? Is there legal risk? This is the only risk, for the most part, that I see in small product, as well as inside enterprise—you worked at a bank—inside the enterprise.

Most of the time, the discussion is more about what’s our risk profile with this? Are we in legal compliance? Are we making money, and are we saving money? We’re not usually asking about because that risk is so far—it’s perceived as being downstream, and we can’t even know if we’re causing it, just—

Ovetta: [crosstalk 00:33:48].

Brian: We got to go fast, so we don’t have time for that stuff. You know, like, to go red team this and think of abusability testing—I love this word [laugh].

Ovetta: Yeah. Abusability, yes.

Brian: We don’t have time for that.

Ovetta: If you don’t have time for that, then you shouldn’t be using AI.

Brian: Yeah.

Ovetta: I’m sorry. I’m just—and this is what I talked about, the two sizes of risk house. There’s the legal, privacy, data privacy, regulation, compliance. EU has a wonderful AI bill that the US is trying to emulate, if ever anybody in Congress really understands what AI is. You know, there’s a new AI group in DC that’s focused on policy there, which is awesome. But there are some real social, human, societal, what I call traumatized outcomes that heighten the probability of human engagement risks with this technology, and you have to pay attention to both sides of the house. You cannot just lean into the legal and regulatory risks.

Brian: Could you define what human engagement risk is, just for my audience?

Ovetta: Yeah. Yeah. So, that’s a term I trademarked, and it’s really about the risks that are non-mathematical or statistical [laugh]. They’re the risks that we bring. And when I talk about human engagement risk, I’m talking about AI and bias, the biases that we can embed into our data, into our models, and into our model—and that show up in our model outcomes.

And here’s an example. In 2017—I think—2017, the Chicago Police Department started using a predictive analytics—crime predictive analytics algorithm. And the algorithm was used, and this is when predictive policing was, kind of like, a thing. It was like 2017, 2019 around George Floyd stuff, all this kind of stuff. Police officers and police departments started using algorithms to predict where protests was going to happen, where crime was going to happen, all that kind of stuff.

They showed up to this young man’s door on the west side of Chicago. He was on the couch. There were three cops there and a social worker. When he opened the door, he’s like, “What’s going on?” His name is Robert Johnson. He was like, “What’s going on?” And they’re like—the social worker is like, “Are you okay?” He’s like, “Yeah. What are you doing here?” The cops were like, “We’re watching you.”

Robert Johnson ended up on this list, and it was called the strategic subject list. And the strategic subject list was generated by this algorithm that was created by this mathematician at the University of Chicago, thinking that he’s doing a good thing, to help the police to predict—it’s like true crime—to predict who would end up either a victim or a perpetrator of a capital crime in Chicago. Robert Johnson ended on this list. Never mind that he had never been arrested before, he had never been in the what you call the system, but he ended up on this list. And there was a series of lawsuits to try to figure out who was on the list.

And there were thousands of people on this list, and they were using data from things like arrest records, social media profiles, like, all these things. And all of that had what I call traumatized data sets. So, they had racial profiling embedded in some of these police records, connection is not causation, like, all kinds of biases were showing up in the data that they were using in this predictive algorithm. And so, a person who was completely innocent of any type of crime is now being watched by the police. And that’s what I mean by human engagement risk.

So, for Robert Johnson, this model was a very high human engagement risk because it brought the police to his door. And he did end up getting shot, he did end up becoming a victim, and he blamed this algorithm. I could go on. Like, Edward French, 71, ended up dead because in San Francisco, they were using an algorithm called a threat assessment algorithm, that the jails were overcrowded, so they’re trying to find ways to let prisoners who were non-violent offenders go. This kid showed up in a courtroom. The algorithm said, ‘low threat assessment,’ but the judge and the prosecutor were like, “I’ve seen that kid before. I don’t think that that’s right,” but they deferred to the algorithm. This is called automation bias, right?

Smart, judge had been on the bench for years, prosecutor years, ignored all of their human knowledge, and went with the machine. Happens all the time. That kid ends up getting released. Two weeks later, he’s on a mountain, robbing Edward French and ends up killing him, right? He never should have been out. They found out later there was wrong data. It was wrong data, and that’s how he got out.

Again, a millionaire gave help to create that threat assessment algorithm for the state. He was trying to do something good, right? But that’s a human engagement risk. And the human engagement risks are biases that intersect with this technology. Automation bias is one of them. Automation misuse is another one. You see this often in pilots. That’s why pilots have to have more manual training, right, because they kind of check out on automation when it’s on autopilot, things like that.

There are things that happen to our brain when we’re exposed to artificial intelligence, and that is research-based. I’m not making that up. And so, because of that, there are real human engagement risks that are opportunity for design. So, when you’re designing a self-driving car, you can’t just let the person go to sleep, unless the car is fully able, and there’s every other car on the road that’s all self-driving. If there are humans behind the wheel, you need to be able to have a feedback loop system, something that’s going to happen because the algorithm is going to be wrong, right?

If you don’t have that designed, there’s going to be a large human engagement risk that a car is going to run over somebody who’s pushing a bike up a hill, which is what happened in Arizona. Why? The car could not calculate the right speed and pace of a person pushing their bike. It had the speed and pace of a person walking, the speed and pace of a person on a bike, but not the two together. Again, algorithms will be wrong, right?

And I remember talking to the chief risk officer at Capital One about an a model that was 98% accurate that he approved, that ended up coming back to bite Capital One. And I said, “You worry about the 98%, but it’s my job to worry about the 2%.” And the 2% is where the human engagement risks happen because you don’t know where the data comes from, right, and if you don’t identify what I call traumatized data sets, so data sets that don’t include populations, right? We just got the first female crash test dummy in Germany in, like, 2021. So, all of that accident stuff and safety stuff was built, not on my height, and body weight, and all that.

Brian: It’s for me.

Ovetta: Right. It’s five nine [crosstalk 00:41:48].

Brian: I’m safe. I’m feeling good, Ovetta [laugh].

Ovetta: Right? If you’re—

Brian: I trust all that stuff [laugh].

Ovetta: You’re the safest person in the world, right? So, things like that. That’s what I mean by human engagement risk because AI can’t live without data, and all of that data includes all of our stuff, good and bad.

Brian: To me, the challenge here—especially, like, in business, when we’re developing products and things like this—is, there’s this space of, like, what is acceptable risk? Is it? Is it zero risk? And you talked about the 98-2, but I think the challenge for product teams is, like, how much resource do we put against the 2? When do we ship this car? When do we ship this algorithm to police departments?

We know it can be wrong because it’s probabilistic. It’s never going to be perfect. It’s only as good as the training set, and we already can identify bias. So that, to me, is maybe one of the hardest questions here for teams, is, what does it mean to have acceptable level of risk? And I find that challenging.

Ovetta: Well—

Brian: It opens up so many conversations.

Ovetta: You need to have diverse people in the room having that conversation. Because again, what we just said, we were laughing about it, but risks to you is going to be different than risks to me.

Brian: Totally.

Ovetta: Right?

Brian: Absolutely.

Ovetta: Because the world—like, again, like, medical journals, research, all that, mortgages, decisions, all that, like, a lot of that data doesn’t include people like me. So then, now you’re using that to create a product that will affect me. So, I better be in a room or represented in some way, to help you decide when the risk is too much. And I think also, again, this is a dynamic product, so when you ship it, it’s not done. So, creating robust feedback loops from diverse populations is one way that you can tell if your model isn’t making the cut. Right?

Also, it’s like, there’s no perfection here. There’s only lines of tolerance and goodness. Model goodness used to be the purview of companies, of the data scientists. This model performed like [crosstalk 00:44:16]

Brian: Accuracy, yeah.

Ovetta: 98% accurate. That is model goodness. We’re letting it go. I mean, if you think about the first search engines, I think their model goodness was like at 77% or something. Now it’s, like, 99%, but it was like, “75? That’s good,” right? Like, you know. And then people started seeing photos of when I typed in black people, of apes, right?

And so, they’re like, “Hey, that’s 77 we got to move that. It has to be more accurate than that,” whatever. So, I think companies have to get used to going to their consumers and customers in a wide spectrum, wide variety, and asking them, when they’re right and wrong. They can’t take on that a burden themselves anymore. And I don’t think you want, to be quite honest.

And again, like, when I speak about this, I know this is hard. This is hard work. This is not easy stuff. And you talked about, nobody talks about the data and all that kind of stuff because it’s hard work. It’s hard work. Having ethically sourced data input and variables, that is hard work. That means you have to do data lineage to know every data point that goes into your model. Again, if you’re going to use this technology [laugh], right, you need to put into place the governance that needs to be there.

Banks have been doing it for 30 years. They’re really good at it. They have really good model risk governance policies, frameworks, and processes. Companies like Google, and Amazon, and that other company I used to work for—Microsoft—are leading with the technology, but they’re fast following up with the guardrails because consumers are going to demand it, right? And I’m not saying I have all the answers. Nobody does, but I am saying you need to seek out these answers about the human engagement risks that can occur with this technology if your company decides that they’re going to use it.

Brian: And part of that is you have to be able to even conceive what could go wrong. And I think part of this is, like, if you can’t even conceive that, there’s no way you can correct for it because you don’t even know it’s a [laugh] problem.

Ovetta: Yeah. Nobody wants to be Negative Nancy. Who wants to do that? Like, everybody, wants to talk about the power of this technology, and I think the power of the technology is awesome, but like I said before, when I’m looking to recruit for my team, I’m looking for somebody who can understand both the capacity and the limitations of this technology, keep those both in their brain as they’re designing.

Brian: How do you balance this with your product colleagues? I mean, I can imagine, there’s we got to ship, we have a release coming up, get this feature out, get this thing out. And these two things are a little bit at odds with each other. How much risk coverage are we going to put in for theoretical things that might happen down the road? That’s an expense. It costs more to do it right. It usually costs more to do it right, whether it’s for user experience reasons or for things like this. I’m just kind of curious, like, how do you have these conversations? How do you get funding to spend time to plant the seeds at the beginning [laugh]?

Ovetta: Well, I’m not always successful, but I think one of the major things I did after I left Capital One, and I started talking about this stuff, was adopting risk management language, taxonomy, and frameworks. Because when you talk to business folks about risk management, they get it. When you talk to them about social, cultural, gender, you know—

Brian: Right.

Ovetta: —it’s not like they’re heartless, but it’s really, you have to make a leap to get to the business case right? But when you talk about risk management, when you talk about the risk of having a model outcome cut off a potential customer source because it’s not inclusive, the conversation changes, right? And this is where [crosstalk 00:48:30]—

Brian: Or a New York Times risk, that’s another kind.

Ovetta: Yeah, yeah. We used to talk about the Wall Street Journal risk, which is what Dov and I were talking about when that 98% went to bite him in the—and they—this was a famous case at Capital One where a woman—Capital One had an algorithm to identify merchants that you were using your card at. Sometimes they use, like, store names and stuff like that—or, I’m sorry, they use, like, numbers instead of store names—so you, like, what did I spend this $3.66, whatever—and so they used an algorithm that used your past purchases, some third-party geolocation stuff, and blah, blah. And they came up with, “Ooh, Starbucks. You spent, you know, 3.69 at Starbucks that day.”

And this woman came home, she was looking at her little Capital One statement, and it said ‘African American surcharge.’ And she was like, “What, what, what? Did you charge me for being, you know, a black woman?” Because that was in her brain. That’s what happened. Well, no, what happened was the algorithm had a null value, and it went to the most popular word, which was American, and the last word, which was some African shop she went to and put it together, right?

So, the algorithm has no idea, right, because it doesn’t understand nuances, it doesn’t understand cultural references, it has no moral code. There’s no, like—

Brian: Yeah.

Ovetta: —you know. And again, limitations. And she sued [laugh]. First she went to American Airlines because that’s where the purchase was, and then it ended up being at Capital One. And it was all over the news, and, you know, whatever. And you know, Dov and I was talking about that—the chief risk officer—and he was like, “Yeah, 98% accurate, that algorithm was.” And I was like, “Yeah, but it’s my job to worry about the 2%.” And he was like, “Yeah.”

And so, now he was like, “Well, maybe I’m not the only person that should be worried about risk,” right? I say that to companies. I use that example as not because Capital One was wrong. It was more like a robust feedback loop from your customers, A, telling them that this result was generated by a machine, and B, giving them a way to tell you when it’s wrong, are, like, must-haves.

Brian: You’re probably aware of the AI Incident Database, Sean McGregor’s work. Are you familiar with that?

Ovetta: Yeah, yeah.

Brian: Yeah, yeah. So, episode 70, I had Sean on. There’s lots of examples of these kinds—if you’re wondering, like, why do we need design in this space, well, you can back-end. Start with the news stories and work your way backwards, and that’ll tell you why [laugh].

Ovetta: When I say, “Human engagement risk,” that’s what I mean, like, these are real risks in different ways, from—like, I said, “You can’t put a price on feeling othered,” because it’s, kind of like, a subjective thing. And Dov was like, “Oh, she put a price on it.” Right? Like, there’s going to be a price there. Yeah, to, like, real, like, data breach, and all that kind of stuff. Like, people say, “Oh, that’s real risk.”

But these are risks too. The idea that if a mother goes to look for jeans for her kid and your AI-generated description talks about dressy, fashionable, flat, whatever, and the mother is, like, really looking for a rugged adventure, you know, jeans that will last, then they’re going to feel left out, right? And that’s a real risk you’re going to lose a customer, right? And so again, like, where does the AI get that engendered language from? Us. So, how do you identify that? Test it. These are things that you have to start doing when you decide you want to use this technology.

Brian: Yeah, yeah. Ovetta, this has been great. There’s another—if you’re interested in law enforcement and analytics, I talked to Allison Weil back episode 42—this is three years ago—actually, this was on the use of predictive analytics inside the police department to predict police officer behavior. And this was actually more the situation, which it’s already validating what everyone knows, but no one really wants to make changes. It’s just validating what people, kind of, knew already, and so that was an example of technically right, effectively wrong.

The solution really did nothing because no one was ready to make a decision in the real world with this information, so it was kind of pointless. It was very, very interesting. But I want to finish up with you here. It’s been great to have you. Is there anything you’d like to just share with my audience that I didn’t ask you about, or just any kind of closing thoughts?

Ovetta: Yeah. You know, I just—thank you so much for this opportunity to talk about human engagement risk and all that. But I do want to say, I am no Luddite, right? I’m bullish on artificial intelligence. I just want us to, for just a hot minute, to be more mindful about how we want to integrate this technology into our society. That’s it. Just take a half a second to think what could go wrong, and how do I use design to minimize that? That’s it. That’s all I’m asking. And I feel like if more of us do that, then some of those really wild things that we just talked about become lore and not our future.

Brian: So, good to chat. This was great. Where can people follow you? LinkedIn?

Ovetta: Yeah, I’m on LinkedIn. Please reach out. I don’t really give my Instagram out because it’s just me and flowers. I like flowers and stuff [laugh].

Brian: [laugh].

Ovetta: There’s nothing like weird—

Brian: Predictive flowers?

Ovetta: Yeah, predictive flowers.

Brian: Is it going to bloom?

Ovetta: [laugh]. Yeah, LinkedIn is probably the best.

Brian: Okay, cool.

Ovetta: Thanks a lot, Brian, this was great.

Brian: Yes. Thank you so much.

Ovetta: I hope this was useful for your audience.

Brian: I learned a ton, and I started this mostly for me, and hoping it would help someone else, too, and 150-so episodes later, like, we’re still going, so [laugh].

Ovetta: Good job. Good job. I can never stick to anything past 21 days—

Brian: [laugh].

Ovetta: [crosstalk 00:54:59].

Brian: Great. Well, thank you, Ovetta. We’ll be in touch, okay?

Ovetta: All right. Thanks, Brian. Bye-Bye.