Episode Description

Guess what? Data science and AI initiatives are still failing here in 2024—despite widespread awareness. Is that news? Candidly, you’ll hear me share with Evan Shellshear—author of the new book Why Data Science Projects Fail: The Harsh Realities of Implementing AI and Analytics—about how much I actually didn’t want to talk about this story originally on my podcast—because it’s not news! However, what is news is what the data says behind Evan’s findings—and guess what? It’s not the technology.

In our chat, Evan shares why he wanted to leverage a human approach to understand the root cause of multiple organizations’ failures and how this approach highlighted the disconnect between data scientists and decision-makers. He explains the human factors at play, such as poor problem surfacing and organizational culture challenges—and how these human-centered design skills are rarely taught or offered to data scientists. The conversation delves into why these failures are more prevalent in data science compared to other fields, attributing it to the complexity and scale of data-related problems. We also discuss how analytically mature companies can mitigate these issues through strategic approaches and stakeholder buy-in. Join us as we delve into these critical insights for improving data science project outcomes.

Highlights/ Skip to:

• (4:45) Why are data science projects still failing?
• (9:17) Why is the disconnect between data scientists and decision-makers so pronounced relative to, say, engineering?
• (13:08) Why are data scientists not getting enough training for real-world problems?
• (16:18) What the data says about failure rates for  mature data teams vs. immature data teams
• (19:39) How to change people’s opinions so they value data more
• (25:16) What happens at the stage where the beneficiaries of data don’t actually see the benefits?
• (31:09) What are the skills needed to prevent a repeating pattern of creating data products that customers ignore??
• (37:10) Where do more mature organizations find non-technical help to complement their data science and AI teams?
• (41:44) Are executives and directors aware of the skills needed to level up their data science and AI  teams?

Quotes from Today’s Episode

• “People know this stuff. It’s not news anymore. And so, the reason why we needed this was really to dig in. And exactly like you did, like, keeping that list of articles is brilliant, and knowing what’s causing the failures and what’s leading to these issues still arising is really important. But at some point, we need to approach this in a scientific fashion, and we need to unpack this, and we need to really delve into the details beyond just the headlines and the articles themselves. And start collating and analyzing this to properly figure out what’s going wrong, and what do we need to do about it to fix it once and for all so you can stop your endless collection, and the AI Incident Database that now has over 3500 entries. It can hang its hat and say, ‘I’ve done my job. It’s time to move on. We’re not failing as we used to.’” - Evan Shellshear (3:01)
• "What we did is we took a number of different studies, and we split companies into what we saw as being analytically mature—and this is a common, well-known thing; there are many maturity frameworks exist across data, across AI, across all different areas—and what we call analytically immature, so those companies that probably aren’t there yet. And what we wanted to draw a distinction is okay, we say 80% of projects fail, or whatever the exact number is, but for who? And for what stage and for what capability? And so, what we then went and did is we were able to take our data and look at which failures are common for analytically immature organizations, and which failures are common for analytically mature organizations, and then we’re able to understand, okay, in the market, how many organizations do we think are analytically mature versus analytically immature, and then we were able to take that 80% failure rate and establish it. For analytically mature companies, the failure rate is probably more like 40%. For analytically immature companies, it’s over 90%, right? And so, you’re exactly right: organizations can do something about it, and they can build capabilities in to mitigate this. So definitely, it can be reduced. Definitely, it can be brought down. You might say, 40% is still too high, but it proves that by bringing in these procedures, you’re completely correct, that it can be reduced.” - Evan Shellshear (14:28)
• "What happens with the data science person, however, is typically they’re seen as a cost center—typically, not always; nowadays, that dialog is changing—and what they need to do is find partners across the other parts of the business. So, they’re going to go into the supply chain team, they’ll go into the merchandising team, they’ll go into the banking team, they’ll go into the other teams, and they’re going to find their supporters and winners there, and they’re going to probably build out from there. So, the first step would likely be, if you’re a big enough organization that you’re not having that strategy the executive level is to find your friends—and there will be some of the organization who support this data strategy—and get some wins for them.” - Evan Shellshear (24:38)
• “It’s not like there’s this box you put one in the other in. Because, like success and failure, there’s a continuum. And companies as they move along that continuum, just like you said, this year, we failed on the lack of executive buy-in, so let’s fix that problem. Next year, we fail on not having the right resources, so we fix that problem. And you move along that continuum, and you build it up. And at some point as you’re going on, that failure rate is dropping, and you’re getting towards that end of the scale where you’ve got those really capable companies that live, eat, and breathe data science and analytics, and so have to have these to be able to survive, otherwise a simple company evolution would have wiped them out, and they wouldn’t exist if they didn’t have that capability, if that’s their core thing.” - Evan Shellshear (18:56)

• “Nothing else could be correct, right? This subjective intuition and all this stuff, it’s never going to be as good as the data. And so, what happens is, is you, often as a data scientist—and I’ve been subjected to this myself—come in with this arrogance, this kind of data-driven arrogance, right? And it’s not a good thing. It puts up barriers, it creates issues, it separates you from the people.” - Evan Shellshear (27:38)
• "Knowing that you’re going to have to go on that journey from day one, you can’t jump from level zero to level five. That’s what all these data maturity models are about, right? You can’t jump from level zero data maturity to level five overnight. You really need to take those steps and build it up.” - Evan Shellshear (45:21)
• "What we’re talking about, it’s not new. It’s just old wine in a new skin, and we’re just presenting it for the data science age." - Evan Shellshear (48:15)

Links

• Evan on LinkedIn
• Get the Book:
  Why Data Science Projects Fail: The Harsh Realities of Implementing AI and Analytics, without the Hype
  • Use code dspf20 to get 20% off at Routledge.com
  • Buy at Amazon*
    (*Paid Link - As an Amazon Associate, I earn from qualifying purchases)

• Why do we still teach people to calculate?
  (People I Mostly Admire podcast)

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, I have Evan Shellshear on the line from Down Under. How are you, Evan?

Evan: Fantastic.

Brian: Excellent, excellent. So, you pitched me—I think I told you this; I get pitches all the time for guests that want to come on the show, and I’m super picky about that. Most of them, I select myself. And you got through. You might have been the only person out of all the episodes to send me something.

And I think I told you right out the gate, I wasn’t interested in talking about this topic: why data science projects fail. It’s 2024. And the reason why is, like, I’m so ti—I’m so tired of this in some ways because this—I felt like this is the same news, again. It’s 2024; it’s almost like there’s no news because it’s just constant all the time.

So, you wrote this book. The full title is Why Data Science Projects Fail: The Harsh Realities of Implementing AI and Analytics, without the Hype. What is any different than my—and this—by the way, Evan, this was my number one article for web traffic. Not that I—I don’t even care about this stuff, but my number one article was a meta article which just listed every study that I kept seeing about how data [laugh] data science projects fail, another study another survey. And I just stopped updating it because it was just the same news over and over and [laugh].

So, what’s different? It’s 2024, this conversation is still going on. I think things are getting better, even—I’m not a data scientist, right? I tried to serve them my work—I do think things are getting better. And I might be biased because, you know, I run this community now, and there are people having success with this. And I see it at conferences that it’s not so rare now. But apparently it’s still, overall, it’s not [laugh] doing well. What, like, why is this news now? Like, why do we need to talk about this again?

Evan: See, that’s the whole point. It’s almost like, after so long, and after so many frameworks, and after we talked about it so much, at some point, you just hope that something would change, right? You think something’s got to give. Like, we’re all looking at this, and it can’t be we’re falling into the same traps because we know them. We know what’s going wrong, we know what’s happening.

And so, it reaches this point where, okay, it mustn’t be these things we’re thinking of. It mustn’t be these simple things that we’re finding, all these common factors that we keep pulling out of. It just can’t be that, otherwise we would have fixed this, right? People know this stuff. It’s not new anymore.

And so, the reason why we needed this was really to dig in. And exactly like you did, like, keeping that list of articles is brilliant, and knowing what’s causing the failures and what’s leading to these issues still arising is really important. But at some point, we need to approach this in a scientific fashion, and we need to unpack this, and we need to really delve into the details beyond just the headlines and the articles themselves. And start collating and analyzing this to properly figure out what’s going wrong, and what do we need to do about it to fix it once and for all so you can stop your endless collection, and the AI Incident Database that now has over 3500 entries. It can hang its hat and say, “I’ve done my job. It’s time to move on. We’re not failing as we used to.”

Brian: [laugh]. And I know you’ve actually gone into detail. I remember that the studies that I saw, you know, they would generally list, you know, five to ten reasons, what their leading indicators were. And part of the reason I started my bus—I mean, I’m a designer, a consulting product designer, and I saw a lot of human factors causes mixed in there. Like, the business doesn’t know what it wants, which to me is code for ‘the problem wasn’t surfaced properly’—that’s a human problem—they don’t know how to understand how to use it, the solutions that we make, change in organizational culture and change management is really difficult, data culture, which again, sounds like everyone else is not capable of using the stuff that we [laugh] made.

From a software design perspective, these are human factors problems there. I saw this a lot. Is this what you’re finding in your data? So, you’ve actually dug into this with a more—a research and academic research perspective on this. Why is this still happening, and what are the root causes here? Are they technical? Is it data quality?

I was just at a conference. Data quality is the most important thing, and master data management. If you want to do AI, it’s like all this defensive stuff. And I’m like, you can get all that platform stuff right and still bomb. Even if the data quality is correct, that doesn’t guarantee any solution built on top of it is going to be good. And I understand accuracy of models will be better, but if people don’t understand why the models are going to benefit them, then it doesn’t matter. You’re technically right; effectively wrong. So, you tell me, what did you [laugh] dig up here in your work?

Evan: This is a really good question, and it actually leads to an even bigger question, Brian. And we’re going to come back to that, and we’re going to unpack why this keeps happening, and there’s a deeper reason beneath all this. And you’re exactly right that the reasons that we saw that were most important are these human factors, things relating to the people and the approaches, and not the technical reasons. The odd technical reason did pop up, like not using the right model, mis-specifying the model, it did actually happen. It wasn’t like it wasn’t there, but it was definitely down the long tail. We’re talking less than five to ten percent of occurrences and reasons that were listed by individuals were due to that.

And the big question we set to set out is not just to rehash that dialog that you’ve had, and you totally understand, we wanted to go—there’s a reason why it keeps happening. And when we dug deeper and started discovering those reasons, we saw really what’s wrong and why this isn’t going to change in the next five years, in spite of all the frameworks, in spite of all the effort we’re putting in, we’re seeing it still won’t change, and we now know why.

Brian: I want to know [laugh].

Evan: [laugh]. I thought you’d never ask. I thought you’d never ask. Our hypothesis—and we devote a whole section of the book to this—one of the driving reasons that leads to, kind of, this not occurring well is this, kind of, disconnect in the middle—okay—this disconnect in the middle between data scientists and the people having to make decisions, like the executives. And why this disconnect exists, it exists in two directions.

It exists in one direction where you’ve got the executives and individuals who are not experts in data science, that’s not their job. Someone running a large retail chain with over 10,000 stores across the United States shouldn’t be an expert in data science. They should be an expert in running these retail chains, their supply chains, managing business, and doing what they do. That’s their job. They’re not expected to be a data science expert, okay?

So, there’s going to be a gap from them to the data science people doing what they’re needing to do, and the expectations set there. On the other side of the gap, there are the data scientists themselves and the people building the tools. And the challenge you’ve got with them is they approach their site from just the technical data science. Often they’re not domain experts. They often don’t know and aren’t aware—especially if we look at the people that are just starting—they’re not aware of the domain they’re applying their data science tools to.

And it’s not just that, that they’re not aware of the domain they’re applying their data science to—and this is what we really found at the core of a lot of this—is they’re simply not being trained in the necessary skills. So, one of the things that I’ve often done when I’ve gone around a given guest lectures—and I’d often do this in mathematics departments—and at the end of it, you have people just staring at me with these weird looks. Like, am I speaking a foreign language? Because I’d be telling them, “Hey, you know what course is missing in your data science degree? Design thinking. You guys don’t know how to unpack a wicked problem.”

Often, unless you’re given a data set that’s often clean, in a nice form, and a problem statement of, try and classify the most number of occurrences of something, they’re going to struggle with it. A real life problem doesn’t come to you like that. A real life problem could be, ‘the Red Cross is considering paying people to donate blood. Should they?’ That’s what a real life problem looks like. It doesn’t come as, here’s a cleaned up data set, and can you classify the people that would be willing to be paid for a blood donation or not? It doesn’t look like that.

And so, data scientists, because they’re missing all the skills that surround that, and they’re missing it because if you look at the degrees across the universities, the top universities in the United States—and we did this analysis of the 40 top master data science degrees—there’s often one unit that has anything to do with something non-technical, and that one unit is typically a capstone project or something similar, and why and how are they supposed to learn all the necessary skills surrounding data science in that one project? It just won’t happen.

Brian: I guess, let me ask you this. I don’t hear about, like, finance teams are failing because they don’t know how to serve the business, they don’t have domain expertise, they don’t understand, like, how they need to serve the executive to—you don’t hear that story very often. I don’t feel like I hear it a lot in, like, digital. And I know what it’s like to move around domains. Like, as a designer, I’ve been exposed to a lot of different domains, and that’s partly why I decided to focus on data stuff, so I can carry domain knowledge forward, and improve the work that I do.

But I feel like with data stuff, why is it so extreme in data that the signal about, you know your technical craft so well, and it seems like so little of the surrounding skills necessary to deploy that successfully. Is this, like, very concentrated just with mathematics or software or something? Engineers, I don’t feel like this is much of a problem with engineering. Less so but, like, in my, you know, 25 years of doing this, like, a lot of engineers become product leaders. They under—because they learn the business, and a lot of innovation comes out of the engineering department. I feel like they… it’s different with data. Why is the [laugh] data thing so—is it unique? I mean, is this just me, or is this unique to data?

Evan: I think there are a lot of really interesting factors to unpacking this one. And I think it comes back to, if we think about what are some similar things that sound like the statistics we’re hearing with data science projects. So, let’s take a step back and think about data science projects and the numbers that we cite in the book. And the closest scientific studies that we could find where people that actually done surveys—and properly carried out instead of just kind of gone on intuition—typically talk about 80%. They typically do about 80% of these data science projects failing.

And let’s take that with a grain of salt, and so maybe it’s between 60 and 95, somewhere in there. That’s the kind of failures that we’re seeing. And if we try and think of what are other things that occur to similar failure rate than that, it’s things like startups, things like starting a new business. When you look at the failure rates, that is typically 90% fail within, I think it’s five or ten years. That’s kind of the common statistic that you’ll see.

And I think part of the reason in the data science space why that’s the case compared to your finance, is data science problems are typically wicked problems, they’re very complex problems. They’re not, like, calculate a balance sheet, or do a P&L, or do something like that. And if finance is really just that. And often finance is compartmentalized into that function: calculate me this, this, and this, and you’ll get those answers. And if you get those answers, it isn’t seen as a failure.

But if you tell a finance person, “Hey, here’s what I want you to do. I want you for the next year or two to figure out how much money you’re going to raise. I want you to forecast the exact sales we’re going to get, exactly to the dollar, and I want you to then calculate how much budget I got for people, and these ten other things that are going to go into solving the full financial challenge with running a business,” then it’ll fail. I can tell you right now, at the end of that year, all their numbers are going to be off by a mile. They’re probably not going to be close to what the business did.

And that’s similar with these AI projects. And the reason why that’s true for AI projects, is firstly because of the fact that they’re going to reach into a domain. They don’t sit in their silo. They’ll typically go into, in a retail setting, perhaps the pricing team, or the category management team, or the supply chain team, and they’re going to lean over these boundaries, and touch, and change the way people operate and run their businesses. And the moment they’re doing that, a lot of these teams, there’s an enormous amount of intuitive expertise that exists, and we’re trying to, in some instances, naively replace that with a data science tool.

But if we recognize that’s often impossible, or very, very difficult, we’re trying to augment that intelligence with the data science tool. Now, you’re right into your world. You’re right in the world of design thinking, you’re right in the world of the human factors. We’re right into that space. And as soon as you hit that space, failure rates are skyrocketing.

Brian: I almost don't know what to say because that there’s so many smart people in this space, and I feel like—I mean, when I go to co—maybe it’s selection bias, but, like, people I’ve had on the show, and when I go to conferences and stuff, like, these are wicked smart people, I mean, some of the people, they seem so smart. It’s like, how do you not see this repeated failure that, like, your team is missing the skill set? I find it interesting, I guess, is the word, or shocked that a management couldn’t see this. Like, is it not—

Evan: What is smart? Like, what is—

Brian: Well, different kinds of smart. Well, I know what you’re saying. There’s different kinds of—

Evan: They’re probably technically geniuses—

Brian: Right, but if you’re repeatedly—if there’s a long history of, like, stuff not working out, it’s like, is continuing to do the same thing going to generate different results? It seems like at some point, at some point, a very smart—generally [laugh], someone that’s just highly intellectual would see a repeating pattern here and change something. I mean, I don’t know. Is there—do you have anything to say about this in terms of skill set changes? Like, are teams trying to change this, or are they not even aware that they’re in the bubble? Like, they’re not even aware of this? Like, what’s the state?

Evan: Boy, and that’s one of the things we tried to unpack for once, and actually get into this, and try and understand if people put effort in, and if they really try and understand this and do something about it, do they move the needle? And the answer is yes, right? What we did is we took a number of different studies, and we split companies into what we saw as being analytically mature—and this is a common, well-known thing; there are many maturity frameworks exist across data, across AI, across all different areas—and what we call analytically immature, so those companies that probably aren’t there yet. And what we wanted to draw a distinction is okay, we say 80% of projects fail, or whatever the exact number is, but for who? And for what stage and for what capability?

And so, what we then went and did is we were able to take our data and look at which failures are common for analytically immature organizations, and which failures are common for analytically mature organizations, and then we’re able to understand, okay, in the market, how many organizations do we think are analytically mature versus analytically immature, and then we were able to take that 80% failure rate and establish it. For analytically mature companies, the failure rate is probably more like 40%. For analytically immature companies, it’s over 90%, right? And so, you’re exactly right: organizations can do something about it, and they can build capabilities in to mitigate this. So definitely, it can be reduced. Definitely, it can be brought down. You might say, 40% is still too high, but it proves that by bringing in these procedures, you’re completely correct, that it can be reduced.

Brian: I don’t run a huge enterprise, but if six out of ten of my bets are paying off, especially with something at scale with what data science can do, it’s like, the those odds seem pretty good [laugh] actually, you know? And the real thing is like, it’s not failure. It’s like, are you repeating yesterday’s failure, or did you learn something? In the four out of ten this year, did you learn anything that you won’t repeat those four again? That’s what’s important, right?

So, of course everyone’s probably wondering, people that are running data teams here, what’s different about the mature ones such that they’re outperforming the immature ones? Maybe you should first tell us, you came with a framework. What is your definition of more mature such that we can align that with better performance?

Evan: That’s a really good question. And it really almost comes back, before you can even talk about that, is understanding kind of what is failure, and as first, we should even take a step back and look at that just for a moment, especially from a human perspective. So, if we talk about success and failure, this stuff isn’t black and white. This stuff lives on a scale of abject failure where every single cent was wasted, nothing was learned, no result was produced, and people got fired and walked away, or the business even went broke right the way through to unmitigated success, where not only did we achieve outcomes, we got even more than that, because sometimes that can happen, that you can go above and beyond what you’re expecting from the beginning of the project.

And so, if we start with something like that, and recognize, okay, there’s a long breadth of possibilities here between this unmitigated success and abject failure—and in the middle could be things like, hey, look, we produced a product, its performance wasn’t quite as good as we hoped, but it still gave us some benefit. Then you’re going to start looking, okay, analytically mature companies are going to be companies that are going to be leaning towards that end in more of their projects. Even if they’re not full, full successes where everything went right, they’re going to be leaning towards that way. And we can see in the literature when we’re looking at these organizations, that analytically immature ones typically are ones that really lack most of the requirements to deliver on analytics projects. So, they are things like high quality data, in-house analytics capability, senior stakeholder buy-in.

And what they’ll really try and do is tackle the AI and data sciences isolated activities that aren’t really connected with the organized strategy. So, they could be pet projects, they could be things that people just have thought up in a moment, that they’re creating on a whim, they haven’t aligned it top-down, and they’re just going for it. For analytically mature organizations, they’re ones that really do have that senior stakeholder buy-in. And there is that desire from the top to make data-driven decisions, and to let data tell the story. There’s a strategy to leverage analytics to improve decision-making. They’ve got the right expectations, they’ve got the right human capital, and the technological resources, and the data, and expertise to apply it, and actually realize some commercial benefits.

So, there’s this—it’s not like there’s this box you put one in the other in. Because, like success and failure, there’s a continuum. And companies as they move along that continuum, just like you said, this year, we failed on the lack of executive buy-in, so let’s fix that problem. Next year, we fail on not having the right resources, so we fix that problem. And you move along that continuum, and you build it up.

And at some point as you’re going on, that failure rate is dropping, and you’re getting towards that end of the scale where you’ve got those really capable companies that live, eat, and breathe data science and analytics, and so have to have these to be able to survive, otherwise a simple company evolution would have wiped them out, and they wouldn’t exist if they didn’t have that capability, if that’s their core thing.

Brian: I imagine there’s probably some frustrated listeners who might self-identify as, look, I have really good people, they’re smart. I think I’m pretty smart. My problem is, I don’t have buy-in, I don’t have the resources I need, the business doesn’t know what it wants, or they’re not data-driven in the culture, et cetera, et cetera. That sounds like a lot of ‘them’ problem. Like, if only the people I was serving were [laugh], oh, were better.

So, does your work state anything about how people move from that side over? I mean, my guess is what you have to show some value in the work, you need to do some small projects that show benefits so that you can get trust, and you can get more money, and more resources, and gradually you build the trust, and that’s how you change the culture. But that’s a zero-to-one problem. Does your data say anything about how these teams were moving that direction? Or they were already there when you ran the study, and you don’t really know how they got there?

Evan: Great question. And really, the question comes from whose perspective are we talking about, first of all. Are we talking about the executive? So, is the executive asking this question, or is the data scientist asking this question because the—

Brian: Let’s say it’s a head of data science, or, you know, VP of data products, or something like this, someone that’s really in charge of that team that’s supposed to be putting value into the organization or the software, whatever the product is.

Evan: Yeah, wherever it’s going for. Look, the reality of those really successful cases that we saw—and we talk about that with American Airlines a lot in the book—Doug Gray, my co-author who’s currently at Walmart, and he’s one of the global leaders in analytics in my eyes, the thing about him is, I don’t think everyone’s necessarily heard of him because he’s not out there promoting his AI expertise on LinkedIn like a lot of these overnight AI experts that we see nowadays on LinkedIn and things like that. Like, he’s out there doing the work. And part of him doing the work, he describes these different organizations—and I can describe them as well—but one of them that really sticks to my mind is American Airlines, where we talk about some of the examples in that book, and the amazing success that led that airline to achieve multi-hundred-million-dollar revenue gains and cost avoidance. So, we’re talking phenomenal savings.

And their success really was that top-down, that CEO who recognized analytics from day one. So, if that’s the case, the simple strategy for a VP of data is, first of all, align yourself with an organization that begins that way, okay. And that’s probably a bit of a cop-out saying that. Try and find that company that is going for it and building it. But if you’re not in that company and the executive team is not supporting it, and you’re lacking that support from above, and you’re just seeing all these failures coming down, then it’s exactly to your point that you need to demonstrate the value. You’ve got to prove the value.

And we talk a lot about that in the book about, like, what language do they speak—and we’re coming back to his human factors—like, what ticks their boxes, and what gets them excited? If you walk in the room and talk about how your R-squared on your regression model is fantastic, clearly you’re going to lose everyone. Everyone’s walked out. But if you walk in the room and say, “Hey, I’ve targeted one of the biggest, juiciest problems in this organization”—and we talked about this with American Airlines; it’s a maintenance schedule that they had, it was worth 500 million a year, right? It wasn’t worth 5 million; it was 500 million a year.

So, you’re getting the executive attention, it’s a number that’s going to be relevant for them, and what you’re doing is then you’re doing exactly like you said. So, the executives are going to sign off and say, “Go for it.” What you’ve then got to do is focus on who’s my next stakeholder? Well, my next stakeholder is the end-user. So, what I’m going to do is go down to the end-user and get them some quick wins so they keep supporting this. Because the executive team’s thinking longer-term, and they’re going to support this over the long-term.

So, your next win, you’ve got to get his on the stakeholder. And you’ll get them won, you’ll get them on board, and then they’ll be able to talk their results up through their business line. And now you’re building support across the organization. And by doing that for a problem that’s going to be worth hundreds of millions of dollars and have a material impact on the business, now we’re talking the executive language. But as long as—I think people are picking projects that they technically like, and people are picking projects that they’re comfortable with, and stuff like that, we’re going to keep seeing those challenges and failures.

And in addition, as long as the data science team, the head of data science is just an order-taker and not a contributor to that strategy—and that really comes back to some of that university education, where is that head of data science is in the strategy meeting coming up with insights that say—instead of saying, “Hey, I’m going to build this model or that model”—and I know you’ve talked about this on a number of your podcasts around churn—and we talk about, “Hey, you know what’s going to happen? I’m going to improve retention so that all the amounts of revenue that we’ve been getting from these customers is going to continue at a higher rate,” because it’s always cheaper to keep a customer than it is to acquire a new customer, that’s going to be language that they’re going to understand. They don’t care how you do it; they assume you’ve got the ability, just like the guy who runs a supply chain. They don’t want to know the nitty-gritty inside out of who’s supplying these goods and how you’re doing it. They just see you’re hitting your target and every year, you’re reducing costs by 5%.

What happens with the data science person, however, is typically they’re seen as a cost center—typically, not always; nowadays, that dialog is changing—and what they need to do is find partners across the other parts of the business. So, they’re going to go into the supply chain team, they’ll go into the merchandising team, they’ll go into the banking team, they’ll go into the other teams, and they’re going to find their supporters and winners there, and they’re going to probably build out from there. So, the first step would likely be, if you’re a big enough organization that you’re not having that strategy the executive level is to find your friends—and there will be some of the organization who support this data strategy—and get some wins for them.

Brian: That makes a ton of sense, and I understand all that. Let’s talk about the skills, though. So, let’s say we figure out we do have an ally in the maintenance department at American Airlines. And the old way we do it is that every plane comes in twice a year to get its whatchamacallits oiled and greased up, and it costs a million dollars per airplane to do this service, right? And now the data science team is going to come in, and effectively we’re saying, “No, we’re going to bring in the right place at the right time that needs service based on the routes that they flew, and it’s really polluted over China, but not over Europe, so those planes will come in more often, da-da-da-da-da, we factored all this stuff in, so we’re only cleaning planes that really need to be cleaned and not taking them out of service, which costs money, and we can quantify how many more flights we could have flown and da-da-da-da-da.”

And the maintenance team is like, “What do you know about maintaining aircraft? Are you an aircraft engineer? No, you’re a data scientist, and you’re going to tell me which planes that I’m bringing into my shop for the last 40 years?” Like, bang, we have a rub now.

Evan: Yeah.

Brian: Does your data talk about the skilling in this place? Because I feel like that’s a lot of time—it’s not that the business opportunity can’t be understand, it’s the, what change will be required of the users or the recipients, the beneficiaries of this service, they may not see it as a benefit yet because the change they’re being asked to make in the real world is substantial, either their identity is attached to it, or they don’t believe that it could be that much better, or the data is so unbelievable. Like, “What? I just cleaned that plane last week. Like, why are we bringing this back in? That can’t be right.”

And maybe it’s like, “Well, the model is 96% right, and you’re looking at the one of the 4%.” And it’s like, “Yeah, it’s not perfect.” This skill set to me, is what I feel like so many of the teams are struggling with—again, selection bias; the people that I talk to with my training, maybe a selection bias. I don’t know. You tell me[laugh], what happens at this stage here.

Evan: This is a really, really good point that you’re making. And I feel that again—and I don’t want to get too down on the universities now, because I’m at a number universities, and teach there, and do all sorts of stuff, but it comes back to—and I think I suffer from it myself, having a PhD in applied mathematics, your attitude you develop when you’re learning is one of, the only right thing is objective scientific data. Nothing else could be correct, right? This subjective intuition and all this stuff, it’s never going to be as good as the data. And so, what happens is, is you, often as a data scientist—and I’ve been subjected to this myself—come in with this arrogance, this kind of data-driven arrogance, right?

And it’s not a good thing. It puts up barriers, it creates issues, it separates you from the people. And some of the best stories we tell in this book is where Doug—and even myself; I have my own, but it’s less so because I’ve often worked as a consultant from the outside, and as a consultant from the outside to a company, you’re given these budgets, you have the deadlines, you got to hit all this kind of stuff, so you just don’t have time to do the human stuff you need to do—but we tell a story in there where, in one instance, Doug sat down with the team for over three months. And first of all, learned how to do their job as well as they could, before he wrote a single line of code. Before he did any data science whatsoever, he just sat with him and understood their job.

Now, again, as a consultant, I haven’t had that luxury. Often, I’d need to get in there and figure it out within less than a week, given the budget constraints. But when that occurs, and you have that human aspect to it where you develop empathy and understanding for your end-users, and you build trust with them over that three months, and they trust that you know how to do their job because you’re sitting next to them in the trenches doing it, that then you actually see what the real problems are. Because it’s easy from the outside just to hear what someone’s saying, but that’s often in a lot of these methodologies of asking why five times gets to the core. And if you’re just given a couple days to get a quick overview, you just don’t have that depth of knowledge.

But if you’re sitting with them, and you know their job and understand how to do it, that’s when you can really start leaning with some power. And that’s where they’ll trust you. And some of the stories in the book that we tell, as well, that are exactly along these lines, it’s the end-users talking about, “Hey, we’re going to lose our job. If you do this, we’re going to lose our job.” And what I think that is driven by, this hype surrounding AI where it’s this all-knowing, sentient, sort of, algorithm that can just do everything without any human input.

And both me and Doug reflect in our careers that in absolutely less than 5% of cases were any algorithms of any sophistication—so that’s beyond robotic process automation—ever fully automated. There was always a human in the loop. So, from day one, you had to design your algorithm to cater for that human in the loop, otherwise it was just going to get rejected by that individual, at the end of the day. And so, by having that human in the loop and showing them the new world after this data science, and the power they’re going to have in their hands, it starts empowering them. And in fact, one of the beautiful stories that we tell in there is where two individuals who delivered this data science to—and I know the book is about why data science projects fails, but we didn’t want it to be all gloom and doom; we do tell a couple of success stories—with both these individuals got a promotion because what they’re doing with their data science tools now were what-if analysis. They’re looking further into the future, and are able to answer executive questions quickly.

So, all of a sudden, they became really valuable. So, now they’re getting promotions, and now they’re actually improving their career prospects. So, it was a real win-win situation where initially there was this fear and trepidation—hey, we’re going to lose our jobs—the data scientists because they build trust with these individuals, was able to work with those fears and said, “Here’s what we’re going to do. Here’s where I’m going to.” They worked with them on building it, and then because they were heavily involved in the build phase, which is the right way to do it, they understood that they’re an integral part of this going forward.

Brian: Yeah. I want to ask you a little bit about, you know, if you went into one of these teams, I’m—knowledge about the problem my team has, but were not able to do this, like, this maintenance, like, we crapped out on this maintenance problem that the maintenance people are still doing it the old way, they run the planes in twice a year, they never used any of our work. We’re missing some skills here. What are the skills my team needs to stop having this kind of pattern over and over again? And before you answer that, I want to let you know I looked up this Freakonomics episode.

So, there’s a show I listen to called People I (Mostly) Admire, and it’s Steven Levitz’s show—I love the title, too—he interviewed Conrad Wolfram. The episode was called, “Why do we Still Teach people to Calculate?” And he talked about how Conrad’s trying to change how math is taught in schools because we are teaching highly technical people calculation skills, which is a small fraction of your ‘should we pay people to donate blood or not’ problem. Eventually, there’s some calculation that’s involved, but so much of the problem is not the calculation part. And his argument is computers and calculators are way better at doing this, and we have so many different ways to do calculation now.

You could run a machine learning model, you could use a calculator, you could—teaching someone to hand long division the problem out or run the statistics or whatever it may be, this is not where we need more of this skill. I’m curious if you found anything like this as well, where is the—coming at the school side, like, the education is not in place? And maybe it doesn’t matter. Maybe we should talk more about what do businesses do about it right now because they can’t go back and, like, re-skill from the start? I mean, that’s like more of a mission kind of thing. But does that correlate to anything you found in terms of the root causes way down below, before the team was hired?

Evan: Absolutely. Look, this opens a big topic, and one of the seminal books on this topic is written by Brian Caplan, it’s The Case Against Education—he’s done a TED talk on it—where effectively he rails against so much of what we’re taught just not being connected to reality, right? It’s just not connected. In the data science sphere, I feel that the data science education is less that. I think a lot of the data science techniques you’re taught are relevant, and you will use them, but I think it’s the other side that we talked about, it’s about what’s missing.

It’s about we’re missing the ability to have design thinking approaches and to use design thinking approaches, to track user journeys, to understand how this data science tool is going to probably be used. But then it’s even more complex now because the data science wants to know—often the executive team above them want them to be able to figure this out—what’s the commercial value of this tool? And especially for really complex tools, it can be very difficult. About a month ago, there was this report released by Goldman Sachs, it was on the 25th of June, and they called it, “Gen AI: Too Much Spend, Too Little Benefit?” And in that, Jim Covello, who’s the head of Global Equity Research at Goldman Sachs—and really an expert in the semiconductor industry—just said, “Look, AI technology is exceptionally expensive, and to justify those costs, the technology must be able to solve these really complex problems. And to be able to do that’s going to require a huge breadth of skills.”

Complex problems, he doesn’t mean it just on the mathematics; he means it on the human side, he means it on the business problem side, he means it on all these different parts. And what he finishes that statement with is, “AI is not built to do that.” Which, I know a lot of people would disagree with him about, but a lot of the applications in business of AI really are these narrow AI applications where they’re solving, they’re doing a forecast of sales, they’re trying to predict the risk of readmission into a hospital. They’re trying to optimize supply chains with vehicle deliveries and stuff like that. Like, this is really that kind of narrow piece, but it’s in a complex space.

And so, for the data scientists and the education—just answering that question—it’s not a matter of necessarily teaching them the wrong things and things; it’s not teaching them enough other things to interact with their business colleagues. Now, in terms of the business itself, you’re exactly right in terms of, it’s a challenge: we can’t re-educate everyone. And so, one of the concepts we talked about in the book, and it’s not new, it’s not something that’s that we’ve come up with, but it’s one that’s existed for a little bit, and we’re seeing it more and more, it’s the analytics translator role. It’s a person that either began with a technical background and went into the domain, and really got interested and excited about it, and has that domain knowledge, but with the analytics background, or—and I’ve met both types in my career, both types exist—or people with domain knowledge that have this data science, sort of, driving interest. And often they’re people that, in their spare time at home, are doing data science projects because they get really excites them; they love this stuff.

And so, those people are spanning that bridge and spanning that gap, and they can work between the data science team and the rest of the organization. And one of the things that often I see need to occur to make that successful is this federated Agile Model of using data science in the organization. So, you can have the data science silo that sits in its corner and gets given projects that they work on, completely disconnected from the rest of the business. But you break that apart, you have those data science sitting in all the respective domain units, but then they get formed as a chapter where they can come together in a chapter and share best practices and learning amongst each other for the projects they’re doing across a business. Now, this isn’t new, but this is where we look at these analytically mature organizations, and they’re adapting the structures like that to be more successful because they’re gripping really into that human side of the data science, and they’re understanding how critical that is, and then making sure they’re not leaving that out of focus and not just giving the executive from above, “Hey, we need some AI. Can you do something with GenAI this week?” And it’s like, “Well.”

Brian: Yeah, “You can go to ChatGPT and type stuff in.”

Evan: [laugh]. Yeah, exactly.

Brian: “There’s your answer.”

Evan: “Go for it.” [laugh].

Brian: [laugh]. But just don’t type anything in about the company [laugh] so I’ve talked about this—is a McKinsey created title—I love and support the work, I can’t stand the title, it sounds terrible because it sounds like you do some analytics, and then you have to translate it at the end. It’s like, “Ah, it shouldn’t require translation if it was clear to begin with.” And I know that’s not what the role is, but I think it’s a poorly—it’s not a good title. It just implies that translation is required.

And so, we have this—‘we’—there are a lot of people adopting this data product management, or this product orientation to working, which, you know, comes out of the software world, and you have a product manager whose job is to produce benefits and outcomes. So, it doesn’t matter how those are accomplished, although on a data team, they’re probably going to be accomplished with analytical methods and data science methods, but they only the delivery of benefits, at the end of the day, however you get to that, there’s a lot of similar crossover there. Although soliciting requirements is not—the framing from a product perspective, the product perspective would be really understanding the problem space and treating the requirements as what I call friendly lies. They’re friendly lies, and sometimes they’re accurate, and they’re not intended to be deceptive, but we need to really discover the problem space.

So, did you see any—I’m curious, are you seeing any of that with the teams that are successful? Or maybe they’re unsuccessful using this? Is there any type of training that a team can do if they are in this shape besides hiring really skilled analytics, translators or data product managers? And maybe the answer is yes, you just have to bring that talent in from outside and stick them in the data team? Or were their places upskilling somehow, and what training were they using? Or did you hear anything about these more mature organizations? Where did they get this outside non-technical help?

Evan: It’s so funny you say that, Brian. In the foreword to our book, Tom Davenport wrote the same thing. He wrote, “I don’t like the word translator. Can we call them data product managers?”

Brian: Oh, he did.

Evan: He said—if you read—read the foreword. He wrote it. He writes—I could quote from it—he said, “I’m more receptive to the idea of data product managers than to translators.” [laugh].

Brian: [laugh].

Evan: That’s exactly what you’re saying. You’re on the same page him, one of the luminaries in the field. But it’s a challenging thing, right? Like Brian, think about it from, like, a data product manager, you can’t just walk in the room and say, “I’m going to be one.” Like, it does require training and skills. Like, you can’t just make it up.

And a data scientist, too, if you’ve got no mathematical or technical training, you can’t just go, “Well, I’m going to be a data scientist.” It just doesn’t happen. Like, you need to get these people that genuinely do have those skills, and you’re either going to hire it or you’re going to train it. And when you’re going to train it, the executive team are not going to say hey, “I want my AI model next week.” You’re going to realize you’ve got a gap, and you’re going to take that time to train it.

And a lot of the brilliant companies—and we see it in books like Good to Great from Jim Collins, when he talks about a lot of organizations’ reaction to the internet boom and a lot of the more mature and successful companies that he talks about in that book—and I know there have been a lot of studies afterwards that have kind of discredited some of Jim Collins’ work showing that actually, those companies that he focused on afterwards weren’t super successful going forwards—but one of the points resonates with me there that he just points out, look, those companies that saw the internet and saw its importance that were ultimately quite successful ongoing, didn’t just react to it. They aligned it with their strategy, they devoted the necessary budget, they found the resources and either trained in-house or hired them from outside of the organization, and built from there. So, it’s not a magic bullet. There isn’t a secret thing here. Again, we’re coming back to these human factors, and human factors, typical human factor things are slower to change. But like you said, before, I like that idea of, what did you call them? Polite lies, or this—

Brian: [laugh]. Friendly lies. Requirements are friendly lies.

Evan: Friendly lies. And I assume, like, the part of the way you uncover those friendly lies, you have this agile approach that really elicits what the people really want by showing them, “Hey, use this,” and if they don’t use it, you realize that was a friendly lie, and you realize they need something else, and you got to keep exploring. And so, similarly here, I think, with a lot of these people kind of elements, there’s this need to really understand the complexity that surrounds them, and knowing that you’re going to have to invest in them. They take time, they take effort, they take iterations, and you’re not just going to be able to get it overnight. So yes, you’ll either have to hire it from outside, or yes, you’ll have to train it up. And, yes, to do that, you’ll want to make sure AI is part of your strategy and not just a flash in the pan or some hype you’re reacting to. You’ll want to know, long-term, is this going to add value because I’m investing in this and I actually want to keep these people and drive this strategy forward.

Brian: Do you think the—and I don’t know if you interviewed or talked to, you know, these VPs and directors of data science and data products that are kind of in that middle level management role on the day-to-day projects and overseeing that work—do you think they are problem-aware such that they know what the skills are they need for their teams such that they can up-level them and get better at this? Or they’re not necessarily aware of what is it, what is the medicine that we actually need? Are they problem aware? Are they solution aware? Like, I don’t know if your data says anything like that, or just anecdotally from talking to these people, but when you talk to them, are you like, “Man, you don’t even see what’s wrong here, do you?” Like, or maybe they were, and you’re like, “Oh, you understand it, and it sounds like you guys are addressing that.” I’m just cur—any stories or just kind of anecdotally—

Evan: So, I get involved in a number of boards and stuff, so—and get involved in a lot of advisory positions. And the problem you are talking about is a fascinating one because it’s just like, how do we go from zero to one? How do you take a company that has no AI skills in the slightest and kick off an AI project? How do you do it? Like, and in particular, how does that organization not get ripped off?

Because they’ve got no AI skills to judge it. They can’t judge a consultant who walks in the door and says it’s a million-dollar project or a hundred-thousand-dollar project. They have no capability of doing that. And so, this is something I’ve discussed with many people and many data product managers. Because to be honest with you, Brian, like, even I myself, I see myself as a product guy.

Like, even though I work in the data science space, I really see myself as a product person. I think many data scientists see themselves as product developers that need to solve these complex problems, and the data science is their tool to do that. Maybe tomorrow, it’s some blockchain thing. Maybe it’s a virtual reality, whatever it is, it’s something to solve that problem. And so, when we look into this, what I’ve understood is, unfortunately for these things that are difficult like this, isn’t some on-off switch, you can just switch and move into it. And these VPs I’ve talked to recognize that.

And one of them told me this story—and this is what I tell everyone, it really resonates with me what he said—“Here’s how I got to AI, and it was a thousand little steps.” And he said, “What I began with is knowing that the organization”—and we in the organization, he wasn’t an AI guy to begin with, okay? He wasn’t a formal AI-trained individual, and what he began with the organization was they wanted to do better, and they knew data was the key. So, the first thing they did was just got their data in order. First step to AI, they just got their data in order. They made sure they had good data governance, and they made sure the data coming through, they trusted.

Once they started having good data in order, they started creating BI dashboards, visualizing that data, playing with that data, started asking questions of that data that they would then validate with the data and their intuition at the same time. So, they’re building up capabilities. You can see this. Then, once they’d done that, they were willing to understand, okay, I know where I am right now, and there’s this little next step that I can take, that I can actually conceptualize, that I can think of. I may not be able to do it, but I’m at the point where I’ve developed enough maturity that I can actually conceptualize what that next step is, and I can get someone to do that and trust that they can do it.

Because I know roughly how difficult it is what I’m doing, and I can kind of roughly guess what that next bit is. I’ve done a bit of reading and thinking about it, so now we’re going to take that step. Maybe we can to bring in someone, or we’re going to train someone up, but they can go to that next step. Now, they might be doing something—you often have this Gartner maturity model that goes from descriptive all the way up to prescriptive analytics, and you’re walking your way up this chain here, step-by-step, building that capability. And I think that’s a challenge in many organizations, so that patience just doesn’t exist.

Knowing that you’re going to have to go on that journey from day one, you can’t jump from level zero to level five. That’s what all these data maturity models are about, right? You can’t jump from level zero data maturity to level five overnight. You really need to take those steps and build it up. And so, with this VP, this was really insightful for me where, I understood how to solve a zero to one problem isn’t going zero to one; it’s going zero to 0.1, to 0.2, to 0.3, to 0.4, to 0.5, walking all the way up, and you’ll get there. If you try and go from zero to one, you’re sitting in that 90% bucket of failures. That’s it.

Brian: Evan, this has been great. I’m really glad that we had a chance to jump into this. So, your book, there’s a good place to get it: it’s routledge.com, right? You have a [cough]—there’s a discount code you were generous to give us, which is DSPF20. So, Data Science Projects Fail, if you’re trying to remember those letters: DSPF20. If you go to routledge.com, that’s R-O-U-T-L-E-D-G-E dot com. That’s the best place to get it. Any closing thoughts for my audience? Any takeaways that you think they should hear about from the book that I didn’t ask you about?

Evan: Yeah look, there’s really—there’s this book out there called—and I love the title; it’s a play on, “To err is human,” and it was written a long time ago, back in the, I think it was in the ’80s. And it was written by a person called Henry Petroski. And Henry Petroski came up with this book title called To Engineer is Human. And he wrote this 40 years ago, right? And he has these comments in it where—that really resonated with both of us—where he showed that a lot of the challenges in these complex engineering projects really come back to human beings.

And he really showed that some of the challenges that we face with the technical problems is this over-reliance on technological tools that leads to people to overestimate how good they are and how much they fail. And it was really an enlightening moment to see that a lot of what we’re doing is just reinforcing the messages that we already know. And in fact, we tried to find something where someone had said this the longest time ago. We wanted to find somebody who really went back into history books, who was really harping on about this a long time ago? And I’m sure we’ve been talking about this for ages, but there was one that was just fantastic, right, and I just have to share it with everyone, and we quote it in the book.

And it comes from Dale Carnegie in his famous book, How to Win Friends and Influence People. And what he says is, investigation and, “Research done a few years ago under the auspices of the Carnegie Foundation for the Advancement of Teaching uncovered a most important and significant fact—a fact later confirmed by additional studies made at the Carnegie Institute of Technology.” And what he says is, “These investigations revealed that even in such technical lines as engineering, about 15 percent of one’s financial success is due to one’s technical knowledge, and about 85 percent is due to skill in human engineering—to personality and the ability to lead people.” This is a hundred years ago, Brian.

Brian: [laugh].

Evan: What we’re talking about, it’s not new. It’s just old wine in a new skin, and we’re just presenting it for the data science age.

Brian: It’s funny, it’s like that Pareto Curve, it’s almost the 80/20, you know? I think everything seems to come back to this 80/20 all the time. Well, I love that quote, and thank you so much for coming on the show. Where can people follow you? Are you on LinkedIn, or if they wanted to get in touch with you, email, social media, anything like that you can share?

Evan: Yeah, look, I’m on LinkedIn. If you look up Evan Shellshear, I’m there. And one of the things I pride myself on, Brian, and I’ve taken this as a role model, I’ve had a few people who’ve been role models to me like this is, when people reach out to me, even if you’re a student just starting your studies, I will respond to you. If you connect with me and send me a message, I make it a point of responding to everyone.

I’ve had so many people give such fantastic things to me along my journey that absolutely, I take every opportunity to give back and share my knowledge, what little of it I do have, with others who are beginning their journey, and especially with something that’s as difficult as data science, right? It’s a challenging field. So LinkedIn, reach out to me. That’s probably the one where I’m there. I’ve got a Twitter profile, but I don’t really use it at all. So, connect with me there, and I’ll absolutely be more than happy to answer your questions.

Brian: Cool. Evan, thank you so much. I’m so glad that you got through the crack, and we were able to do this show and share this knowledge with everybody. And thank you for your work to dig into the facts there and go a little deeper.

Evan: Absolute pleasure. It’s so fantastic, Brian, as a fan of your show to be able to contribute to it, it’s a dream come true. Thank you.

Brian: Oh, good. I appreciate that. Thank you so much.