I keep hearing data product, data strategy, and UX teams often struggle to quantify the value of their work. Whether it’s as a team as a whole or on a specific data product initiative, the underlying problem is the same – your contribution is indirect, so it’s harder to measure. Even worse, your stakeholders want to know if your work is creating an impact and value, but because you can’t easily put numbers on it, valuation spirals into a messy problem.

The messy part of this valuation problem is what today’s episode is all about—not math! Value is largely subjective, not objective, and I think this is partly why analytical teams may struggle with this. To improve at how you estimate the value of your data products, you need to leverage other skills—and stop approaching this as a math problem.

As a consulting product designer, estimating value when it’s indirect is something that I’ve dealt with my entire career. It’s not a skill learned overnight, and it’s one you will need to keep developing over time—but the basic concepts are simple. I hope you’ll find some value in applying these along with your other frameworks and tools.

Highlights/Skip to

• Value is subjective, not objective (5:01)
• Measurability does not necessarily mean valuable (6:36)
• Businesses are made up of humans. Most b2b stakeholders aren’t spending their own money when making business decisions—what does that mean for your work? (9:30)
• Quantifying a data product’s value starts with understanding what is worth measuring in the eye of the beholder(s)—not math calculations (13:44)
• The more difficult it is to show the value of your product (or team) in numbers, the lower that value is to the stakeholder—initially (16:46)
• By simply helping a stakeholder to think through how value should be calculated on a data product, you’re likely already providing additional value (18:02)
• Focus on expressing estimated value via a range versus a single number (19:36)
• Measurement of anything requires that we can observe the phenomenon first—but many stakeholders won’t be able to cite these phenomena without [your!] help (22:16)
• When you are measuring quantitative aspects of value, remember that measurement is not the same as accuracy (precision)—and the precision game can become a trap (25:37)
• How to measure anything—and why estimates often trump accuracy (31:19)
• Why you may need to steer the conversation away from ROI calculations in the short term (35:00)^[e]

Quotes from Today’s Episode

• Even when you can easily assign a dollar value to the data product you’re building, that does not necessarily reflect what your stakeholder actually feels about it—or your team’s contribution. So, why do they keep asking you to quantify the value of your work? By actually understanding what a shareholder needs to observe for them to know progress has been made on their initiative or data product, you will be positioned to deliver results they actually care about. While most of the time, you should be able to show some obvious economic value in the work you’re doing, you may be getting hounded about this because you’re not meeting the often unstated qualitative goals. If you can surface the qualitative goals of your stakeholder, then the perception of the value of your team and its work goes up, and you’ll spend less time trying to measure an indirect contribution in quant terms that only has a subjectively right answer. (6:50)
• The more difficult it is for you to show the monetary value of your data product (or team), the lower that value likely is to the stakeholder. This does not mean the value of your work is “low.” It means it’s perceived as low because it cannot be easily quantified in a way that is observable to the person whose judgment matters. By understanding the personal motivations and interests of your stakeholders, you can begin to collaboratively figure out what the correct success metrics should be—and how they’d be measured. By just simply beginning to ask and uncover what they’re trying to measure, you can start to increase your contributions’ perceived value. (17:01)
• Think about expressing “indirect value” as a range, not a precise single value. It’s much easier to refine your estimate (if necessary) once a range has been defined, and you only need to get precise enough for your stakeholder to make a decision with the information. How much time should you spend refining your measurement of the value? Potentially little to none—if the “better math” isn’t going to change anyone’s mind or decision. Spending more time to measure a data product’s value more accurately takes you away from doing actual product work—and if there isn’t much obvious value to the work, maybe the work—not the measurement of the work—needs to change. (19:49)
• Smart leaders know that deriving a simple calculation of indirect contributions is complex—otherwise, the topic wouldn’t keep coming up. There is a “why” behind why they’re asking, and when you understand the “why,” you’ll be better positioned to deliver the value they actually seek, using valuation measurements that are “just enough” in their precision. What do you think it says to a stakeholder if you’re spending an inordinate amount of time simply trying to calculate and explain the value of your data product? (23:22)
• Many organizations for years have invested in things that don’t always have a short term ROI. They know that ROI takes time, and they can’t really measure what it’s worth along the way. Examples include investments in company culture, innovation, brand reputation, and many others. If you’re constantly playing defense and having to justify your existence or methods by quantifying the financial value of your data products (or data product management team, or UX team, or any other indirect contributor/contribution), then either your work truly does lack value, or you haven’t surfaced what the actual success metrics and outcomes are— in the eyes of the stakeholder. As such, the perceived value is “low” or opaque. They might be looking for a hard number to assign to it because they’re not seeing any of the other forms of value that they care about that would indicate positive progress. It’s easier to write [you] a large check for a big, innovative, unproven initiative if your stakeholders know what you and your team can accomplish with a small check. (35:16)

Links

• Experiencing Data: Episode 80 with Doug Hubbard

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, how to quantify the value of your enterprise data products, your data product management function, and UX team, and why this is not a math problem.

So, what’s driving this topic today? I keep hearing that data products, data strategy, UX teams, particularly those that are working internally inside a large organization, struggle to quantify the value of their contributions, whether it be the team as a whole or on a specific data product initiative that they might be working on. So, while the disciplines here, your job titles and disciplines are a little bit different, the underlying problem is still the same. So, with the exception of those of you working in commercial software, where ultimately, payment for your product, revenue coming in, is the ultimate binary measure of whether value is probably being created, at least on the perception of the business value, if your work is more indirect, it’s harder to measure. And your stakeholders want to know the direct cost savings or revenue impact that your contributions are having, and you probably find it hard to put numbers on this really easily. And guess what? They also are having problems putting numbers on this easily.

So today, one thing I’m not going to talk about are the really easy valuation scenarios out there. A good example of this is something like running an A/B test on the website, maybe roll out a new feature that uses, you know, machine learning to do dynamic pricing, and then on the B side of the test, you just use your traditional human-based pricing, and then you can look at what the impacts are of that particular data product that was deployed, or is some scenario like that. These are easy school-like problems. They’re neat problems, the variables are all known, it’s fairly controllable, and it’s easy to measure. This is not a messy problem; this is pretty nuts and bolts stuff, especially for you analytical types out there.

You don’t need help measuring those, so I’m going to put those types of valuation scenarios aside and really talk about the messy aspects, and kind of the art of figuring out how to estimate value. That’s really what today is about. And I do see this as part analytical, but also there’s a creative and almost an art to doing some of this type of work.

So, this messy problem, right? I almost think of this as less subjective and more subjective. As a consulting product designer myself, estimating value when it’s indirect is something that I’ve had to deal with for many years, and this is not a skill that I learned to do overnight. And it’s really true, whenever your work is part of the whole, you have a contribution that’s part of a whole—and this is very true with design because a design that never gets implemented and pushed out to a customer essentially has no value at all, except in perhaps helping envision what could be or perhaps shouldn’t be. Beyond that, it really doesn’t create any value until it’s implemented, so you have to say, well, then the implementation has some value as well on the engineering side, and what about sales and marketing, et cetera, et cetera?

So, learning to quantify indirect value is a skill that you really need to develop over time. And this is something I always do before I engage in any consulting engagement with a client is, we have a discussion of what the possible value is that we can create together. And this is good for them and for us. I want to know what success means for them, what their expectations are, what we could do together, and I want to know this as well so I can be successful or I can turn down the engagement if I don’t feel like we’re aligned on that, or their expectations of value are well in excess of what I think I could deliver to them with low risk. Having this discussion lowers the risk for all parties, and it really helps us work together.

So, you can leverage some of these same skills. I just wanted you to know where I’m coming from with this. I’m really teaching you some of the skills that I use in my consulting work, but a lot of this stuff can translate to when you’re in an employment role as well. I didn’t use these back in the time when I was still doing W2 and employment work, and I wish that I had built these skills and deployed them. I think I would have been more effective in my roles had I done that at that time.

So, let’s jump in here, just on, like, some of the k—really, I just have a bunch of different key ideas that I want to get across here. The first one is that—again—value is subjective and not objective. So, what do I mean here? If you’ve ever paid $20, $25 for a beer or soda or drink at your favorite sporting or concert event, and maybe that drink is half as much at the bar down the street, and maybe a quarter the price in a grocery store, then you’ve experienced how value fluctuates, and it’s really in the eye to the beholder what it’s worth. So, you know, there’s people that spend $25,000 even $70,000 on stereo cables. Like, if you have a Hi-Fi, you know, stereo receiver at home and you like to listen to music, there’s people that spend that just on the cables. In fact, they actually call them interconnects at that price point because that sounds better than cables [laugh].

And there’s other people that might spend 10 or 50 times that much on a handbag, for example. And then there’s other people that are interested in spending their money to get their name on a library at a university or a hospital. And you might be one of these people, you might think that these things are totally insane, and I wouldn’t even spend zero dollars on a handbag, let alone $1.5 million. Either way, you just mentally assigned evaluation when you heard me talk about these scenarios.

So, who’s correct? Well, nobody is and everybody is. So, you should assume your stakeholders are also doing this, particularly when they cannot just easily look up a simple financial metric that will tell them, what’s the value of this product team or project? So, where do we go next with this? Well, the next big topic here is that measurability does not necessarily mean value. Not everything that a stakeholder values may need to have a currency or monetary value assigned to it for you to show impact.

Even when you can assign a dollar value to something fairly easily, the data product you’re building or whatever it may be, that does not necessarily represent the facts about what a stakeholder actually feels about it. A simple example, you know, for data people out there, you might create a AI-powered dashboard that does fancy forecasting, and it helps the senior leader do their sales work better, or have a better sense of the sales for the organization, or something like this. However, even if that’s—we could argue that this helps, you know, say, gain a million dollars a month and increase sales, what you might not know is that this leader believes that in simply improving on what we were doing yesterday isn’t going to save us from the existential threat we face right now, such as, oh, our competitors are using AI, and surely they’re accelerating faster, and they’re using AI in their sales pipeline somehow. So, in this case, it might be true that while there is that $1 million in increased sales potential that really is measurable, and we can attribute this back to this data product that was built somehow, that may not necessarily be the most important thing to that stakeholder.

So, by connecting with that stakeholder on a personal level, we might find out that what’s keeping them up at night is seeing some type of observable traction on some new initiative, even if that initiative is not revenue positive yet. For example, maybe the company’s moving its services or products into a new market or a new country, or maybe they want to use AI to automate parts of their service or product in order to ensure that they’re staying strategically positioned for tomorrow. By helping that stakeholder understand, what would you need to observe for us to know that we made progress on this initiative or product, and then delivering results against that objective. Well, this positions you as more of a strategic thinker, and it gets you invited to the adult table at Thanksgiving, or whatever holiday you might celebrate that involves status-based dining tables. So again, just to re-summarize that, while we might have an easy to measure initiative, and we can put a financial metric on that, there may be other goals that are less measurable, but are more important to the stakeholder, and if we can dial into that, then the perception of the value of your team and the work that you’re building may simply go up, and there may need less time justifying or trying to pull concrete numbers out of something that is fairly indirect.

The next idea: businesses are made up of humans, so most B2B stakeholders aren’t spending their own money when they’re making business decisions. So, this means that there are more forms of potential value to create for or with these stakeholders, including for example, does the data product increase the stakeholders influence in the organization? Does it make their job or their constituents’ job or the target user base that they’re in charge of overseeing, simpler, safer, less risky, more predictable? And just a quick note on risk, look at the insurance industry. This is a very successful industry, and they’re selling risk reduction, and so in a lot of ways, consumers never use their insurance, yet countless people in orgs buy it because it’s perceived as having value.

So, think about the ways that status quo may increase risk in your organization, like doing what we did yesterday, and doing that again tomorrow, and again tomorrow, and again tomorrow. There can be risk in doing that, and your stakeholders may be feeling that risk, that we can’t just keep repeating what we’re doing tomorrow and making micro, little changes to our products and services because disruption is around the corner, or there’s new competitors coming to the market all the time. We keep hearing about AI all the time. This is definitely on the minds of stakeholders right now.

Other forms of value. Does it give the stakeholder a win that they can broadcast out? For example, can they say, “Hey, this is our first foray into”—fill in the blank—“And we can now show that this team can deliver on this.” Does it align with their personal values? For example, “Hey, any AI that we develop for our organization will be transparent, interpretable, and explainable, as this is ethically the correct thing to do.”

Now, maybe the AI solution that the data team rolled out had very hard or fuzzy, indirect value that was hard to calculate, but maybe in the past, you were using neural nets and black boxes, and this was a real challenge that added risk to the organization because nobody could explain what was going on, and you’ve since transitioned over to this more transparent and interpretable direction because that’s what the leader values and thinks is the right thing for the company, then you are creating value there. It’s just not only in the financial, measurable category; you’re actually helping this stakeholder achieve part of what their greater mission is there, and that’s something that’s harder to assign a dollar value to, but it does not mean that it’s not valuable.

And finally, does the value of your initiative product team, is that helping the org keep up with the Joneses? So, maybe that’s an American reference, but keeping up with everyone—what all the competitors, everyone in the industry is doing. Right now, like, in data product land, it seems like everyone’s trying to adopt a product model, and so we need to do the same to retain or attract the best talent because that’s how, you know, data teams want to work now. Or AI-driven, dynamic pricing is now an industry standard, and we can now say that we’re on par with our competitors.

Now, is the pricing actually performing as well as we hope? Well, maybe it’s not quite performing financially where it needs to be, but if that leader believes that we will be dead in five years if we don’t move to this, then you have helped that leader fulfill one of their objectives. And the higher up you go in the organization, generally, the longer-term the outlook is, and the more they’re thinking about tomorrow, and not just what’s shipping next month, there—hopefully; and I know there’s exceptions, and maybe some of you are like, I wish my [laugh] leaders were thinking that way. But in theory, the captain of the ship all the way at the top really should be thinking about the trajectory of the organization going forward. And I think most people do have both short and long-term objectives, particularly in senior roles.

So, effectively here, you need to dial into what these other forms of value are and realize that they’re not all going to be directly quantifiable dollar figures, and that’s okay because companies spend money on this stuff, and we’re going to talk a little bit more about that later. But moving on to the next one, quantifying a data product’s value starts with understanding what it’s worth measuring, in the eye of the beholders, not math calculations. I’m going to say this again, quantifying a data product value starts with understanding what is actually worth measuring, in the eyes of the beholder, and it does not start with math calculations. So, if you’re in the data and analytics or data science space, you’re likely heavily biased to see this as simply a matter of uncovering the right KPIs, and then running the arithmetic, frankly, to calculate a value. The actual problem here is much more about figuring out or helping to extract the metrics by which the stakeholder is going to assign value in the first place.

So, for example, happiness, as self-reported by a department or team who is going to use the data products your team is building, that might be one of the stated success metrics of the stakeholder. Why? Well, we might find out through questioning that the stakeholder is tired of that team’s complaints, or requests for more resources, or because they see that team as being strategic, and they want to make sure that team has everything it needs to be successful, and they can’t say they didn’t get. So, if I can make that team happy, we’re pretty sure that they’re going to be enabled to do their job, and that’s going to ultimately serve the organization the way we want the organization to be served with their work.

So, how will happiness be measured? Ask, and then deliver on that. So, sorry, that PhD in math that you might have spent a lot of time and money to get might not be needed for right now, but you might find your job security going up and your title changing when you start orienting around what value actually means, and realizing that a lot of this is not about math questions. And again, the best part of this whole topic right now that I’m telling you about is that I think simply beginning to think this way by asking the right questions starts to increase your actual perceived value automatically. Just asking these questions and beginning this process is going to set you apart from people that jump into delivery work immediately.

They don’t know what the desired outcomes of the organization are, they don’t know anything about the personal goals of the person that’s requesting this work, they don’t know what’s in it for them, they don’t know what the risks are. Just beginning to ask this is going to show—you’re effectively showing empathy that you actually care about a stakeholder’s own success. And guess what? Everybody is self-motivated and self-interested to a degree, right? People want to increase their salaries, and their bonus potential, and their job titles, and their status, and things like this, and helping others achieve those goals, well, guess what, it looks good on you as well.

The next topic here is that the more difficult it is to show the value of your product or team in numbers, the lower that value is to the stakeholder, at least initially. I’m going to say this again. The more difficult it is for you to show the value of your product or team in numbers, in monetary units of some kind, the lower that value likely is to the stakeholder, initially. This does not mean the value of your work is low. It means it’s perceived as low because it cannot be easily quantified yet in a way that is observable to the person whose judgment about it matters.

So again, by understanding the personal motivations and interests of your stakeholders, you can begin to figure out what the correct measure should be, together. And chances are, the numbers begin to go up when you help discover what the right metrics are in order to measure this value. And that’s the irony. By just simply beginning to ask about this and uncover what should be our measures, our metrics here, in order to figure out if we’re creating value or not, by going through that process, your value is likely going up already, and now that you have some things to measure, you can start to increase that.

The next one: by simply helping a stakeholder to think through how values should be calculated on a data product, you are likely already providing some additional value. So, we’ve kind of talked about this one, but I think it’s really important to state that the act of you going through a value estimation process or figuring out the metrics by which to measure value, even if you don’t actually execute any calculations yet, you’re actually providing value just doing that. You’re building a relationship, and you’re showing that you’re interested in helping that stakeholder achieve their goals. And the hard part is often figuring out what I call the zero-to-one problem. How do we get from totally vague about what the value is, to some measurement, some estimate that we can observe and then calculate?

And once you’re in that range of being able to observe and calculate, you’ve made the whole valuation thing a lot easier to see. Now, you may invite corrections and see more engagement from stakeholders who might see your math as being off, or that you didn’t factor in X, Y, and Z, and you didn’t do this, that’s all okay because now you’ve gotten out of the zero to stage, and now you’re in the one-to-two stage. You’re making things better together, and they’re engaged in the process, and you’ve got multiple brains now working on this better. So, simply by stating, “Okay. Before, we had no idea. Now, we think a rough estimate is based on this and this and this measurement.” And as soon as you say that, you’re going to start getting that feedback there, and I kind of see it as, like, a snowball effect, right? You’ve finally got the snowball moving, and it can start to grow on its own.

The next one: focus on expressing an estimated value that is a range versus a precise single value. Again, think about expressing value when it’s indirect as a range of values and not a precise single value. It’s much easier to get narrow once there’s a range that’s been defined. And you only need to get precise enough for somebody to make a decision with the information. And it also reduces the chance that people will challenge your math or your thinking or your trust because a range inherently shows, “Hey, I’m not sure. I’m not sure exactly what the value is, and I’m not sure how we’re all thinking about calculating this, but I think this is a good estimate. Do you think it’s good enough to make the decision that you need to make?”

So, maybe you might say, “Hey, we think the value was between $10 and $22 million, depending on how you calculate it.” Now, they’ll likely ask you how you got those numbers. Well, now you can show them your measurements. And ideally, these are measurements that were predetermined at the start of the initiative, or the quarter, or whenever you do your planning; you would have had that conversation at the beginning and not the end of the initiative. You want to know what the measures are up front.

Are they going to poke at it? Probably. Is that okay? Absolutely. You want them to poke at it. You can continue to ask them how much it’s worth continuing to refine the measurement versus doing real work because spending more time to measure these things more accurately takes us away from doing actual product work. So again, focusing on a range and figuring out is the range close enough to make a decision, I think that’s typically much more important than it is coming up with an exact dollar amount.

So, you might say, “Hey, the value is between $10 and $22 million. If you need a single digit, why don’t we pick the midpoint? Does $16 million feel about right to you or do we need to refine this further?” And if they had no idea whether it was $50,000, $100,000, $1 million, $10 million, $100 million, like, they literally have zero idea and it’s so opaque, if you can get it to 10 to 22 and if the average of those two in the middle, the midpoint makes sense, that might be enough for that person to make a decision. Now, obviously it really depends on, well, what are they trying to use this number for? When we talk about value, what’s the number going to be used for? A buying decision, a staffing decision, a resource allocation? That’s part of the conversation. You need to understand that in order to know how specific do we need to get.

The next one: measurement of anything requires that we can observe the phenomenon first, but many stakeholders won’t be able to cite this to you without help. I’m going to say this again. Measurement of anything requires that we can observe the phenomenon being measured first, but many stakeholders will not be able to cite what those phenomenon are to you without your help. You can measure both qualitative and quantitative aspects of something to determine its value. So, when you hear goals, like, “I want confidence that our strategy to use AI this year to achieve X is on track,” well then you need to help define what on track or off track looks like to that stakeholder, and that’s not an objective thing. That is a subjective thing.

The more you dig into this, the more the stakeholder is going to realize you’re not just there to write code, build apps, deploy models into the cloud, build product processes, et cetera, but rather to ensure that technology is creating benefits and outcomes. Smart leaders know that deriving a simple calculation of indirect contributions is complex, and no matter what number you come up with, someone could probably argue that your set of variables or inputs is wrong. But what is not wrong or debatable is the stakeholder’s feelings about the situation. So, if they feel like hearing positive feedback every month from the whatever team that your data products are helping them be more effective at their work, then frankly, I would feel like this is worth continued investment. That’s what I need to hear.

So, you might take that a little further. How much feedback, from whom, and what if X happens instead? These refining questions are helping you narrow down the measurement space and get more specific. And you might get this down to, hey, there’s three department heads. If I hear roughly once a month, or at our monthly check-ins, that they are feeling like their teams are being really—their progress is being accelerated with these data products, they feel more confident in their forecasting decisions, or blah, blah, blah, that might be it.

And the cool thing is, this takes a lot of stress off of you, the producer, to spend all this time trying to quantify in dollars the indirect value, and instead focus on progress, in the eye of the beholder, which might simply amount to ensuring that those three department heads are getting positive feedback from their staff who are using the data products on a monthly basis. You now know exactly what you need to do to make your stakeholder happy, but you need to be able to observe the phenomenon. They need to be able to express, when they say, “I want confidence,” you need to dig into, “Well, how would you know? What would make you feel confident or not confident?” And sometimes exploring the negative side of that, “Show me what un-confident looks like. It sounds like in the past, you’ve not been confident about stuff. What did that look like?” They are going to tell you what that looks like, whether it’s qualitative feedback or things they’re observing in the organization, whatever it is, you’re going to get some signals. And if you have signals, then, by definition, they’re observable and therefore they’re measurable.

So, moving on to the next one, when you’re measuring quantitative aspects of value, remember that measurement is not the same as accuracy or precision, so don’t get pulled into an accuracy debate. I'm going to say this again. When you’re measuring quantitative aspects of value, like dollars and cents or euros or whatever your currency is, remember that measurement is not the same as accuracy or precision. Don’t get pulled into an accuracy debate. So, why is evaluation number needed at all? How is it going to be used? Do you even know?

Like, for example, I say that I’m six feet two inches tall when I’m asked about my height, but I’m actually probably more like six foot 1.92198182112100374 inches tall. And right there, that number is also probably not precise enough if you really want to know exactly how tall I am. Am I even tall? Well, relative to my toddler, yeah. Relative to the average height of an NBA player in the National Basketball Association, well, probably not.

What decision do we need to make with my height? This is where my—I love all y’all analytical types out there, math people—this is not about doing the math well. This is about living in the gray area. It’s about needing to be good enough to help make the decision at hand. Leaders, they’re already making decisions all the time without data or facts to support them, just feelings.

They use the facts and data to support it after the fact, half the time. So, overdoing it on precision isn’t likely to make a big difference, if any at all, and you can end up spending a ton of time on accuracy on something that’s not actually that important. Now, I’m not talking about things like the accuracy of data in the reports that you’re building, I just want to be very clear. This is about assigning value to the work that you’re doing. That’s very different than, like, making sure the financials are correct and all this kinds of stuff.

There’s some similarities here, though. It’s really understanding about how precise do we need to be, to be useful. We don’t want to look like we’re wasting time—or you don’t want to look like you’re wasting time—because you’re spending more time trying to measure the value of your data products versus actually making them better, getting feedback, rolling out changes, listening to customers and users, making things better. So, keep that in mind.

Let’s jump into example on this one. Let’s say your AI enhancements launched January 1st last year, and resulted in an increase of sales of, say, $10 million over the original $30 million forecast. Well, that’s about a 1/3 or 33% increase, right? So, now you’re at $40 million in sales for the prior year. Now, the AI team, whatever the enhancements they launched into the business that helped drive this additional $10 million revenue, well, they probably can’t take all the credit because product management and UX wants some credit, engineering and data science wants some credit, sales, marketing.

Well, there’s five teams just right there, and they all want some credit for their efforts too, on packaging up this new AI-powered feature or product, or whatever it was that contributed to this additional $10 million over the forecast. And of course, it was a chat bot, right? Because that’s what everything is, is a chat bot. So, you might estimate that, let’s say $2 million of that $10 million was attributed to the data team, the AI or data product team. Well, how did I get to $2 million from this $10 million? Well, it’s $10 million divided by five teams. So, let’s say that of the five discipline groups in the product org, well, they each get a $2 million share of that $10 million pie, $2 million worth of credit.

Now, is this correct? Of course not. It’s definitely not to some people, but the real question is, is it right enough, correct enough for the person who’s asking? Their question might be something like, “Look, John Doe, Mary, whoever you are, I’m spending $750,000 a year on my AI team, and I’m being asked to cut back if I’m not seeing a positive ROI. I need to know if this work is producing value that I can justify upstairs.”

So now, if this leader is now getting roughly a 3X return on their investment—why is it 3X? Well, remember, the AI team is getting a $2 million share of credit for that increase in revenue, they cost $750,000 a year. That’s about 700,000 times three is about $2 million, so they’re getting roughly—for every dollar they’re spending, they’re getting three. That might be all they need to know to keep staffing levels the same, or for that stakeholder to make an argument to leave things alone, or maybe even get more resources. If they’re getting a 3X return, they might say, “Hey, that’s fantastic. This is a brand new initiative. We didn’t even know if it was going to work. I can go to bat and get us more team, more resources, more compute power, whatever it is that data team, or the data product team or the AI team needs.” That’s it.

It’s not a perfect number, but it doesn’t need to be. It just needs to be good enough, believable, and ideally, something you worked with that stakeholder on, something that’s enough for them to make that decision. So, don’t get pulled into the accuracy debate because that’s really not what the point is. And finally—well, not finally, we have a couple more here—sometimes an estimate is enough, and you can improve your estimating skills by letting go of accuracy and learning to measure anything. I’m going to say this again, an estimate is often enough, and you can improve your estimating skills by letting go of accuracy and precision, and learning to measure anything.

And my guy for this is Doug Hubbard. Go back to episode 80 of this show, Experiencing Data. I think his book is called How to Measure Anything. It’s a great read. The first half of the book, no math. Second half gets into statistics and how to actually perform calculations, but the real meat and potatoes that most of this audience listening to this show needs is the first half of that book.

And the example that I remember from this about how to measure anything that really made me stop, just stop in my tracks, and was like, wow, my—I just—my head was, like, blown for a second. Was this idea that some things are not measurable and, like, there’s really no way to put any type of value on this, et cetera. The scenario that he talks about is something to do with measuring the wingspan of a modern airliner, if I remember correctly. It’s been a while since I read it, but I’m going to, I just kind of wrote out a little scenario to explain, how do you measure anything? So, let’s say I’m randomly going to ask my spouse, how long is the wingspan of a modern airliner?

Now, my wife would likely say, “You’ve already lost my interest.” And if I was to probe then I’d probably get her to say, “You know what, I have no idea.” Really? Really, she doesn’t have any idea? So, maybe I throw out an estimate. “Is it a million feet long?” Guess what they’re going to say? “Of course, it’s not a million feet.” “Is it 100,000 feet long?” I might say. And she’s probably looking at me like I’m nuts, and like, “No, no, no, no. It’s probably, like, a fraction of that.”

“So, maybe it’s like 100 to 1000 feet long?” And at this point I might get, “Yeah, it’s probably something like that,” and probably a little bit frustrated with the questions. I might want to kind of understand the bottom end of this range, so I might say, “Hey, you know, I’m six feet tall. Do you think it’s like maybe ten of me stacked back to back?” So, that’s 60 feet.

And if she said yes, then now I’ve gotten the original response from her, which was, “I have no idea,” down to an estimate of 60 feet to 500 feet long. Where did I get 500? It’s in the middle of 100 to 1000 when she said, “Yeah, it’s a fraction of 100,000.” You can decide how far to take this exercise, but I think this skill is widely underutilized because we are trained in school to focus on getting the right answer, and there’s no right answer here; there’s just a good enough answer. So, there’s a little bit of an art to this, but it’s actually fairly analytical how to go through this. You’re really throwing out an anchor.

And it’s fun. You can throw out kind of ridiculous anchor at first, and then you can start to learn real quickly that this person you’re nudging them in to thinking about this problem as not being a problem that’s so hard to get from zero to one to, but you’re making it much easier to extract this information out of them. You’re not telling them the answer. You’re actually throwing out the things that get them to respond to it, and you’re now narrowing this down into a range. We talked about ranges already.

So, learn how to measure anything and learn how to estimate. I think this is a really great skill in doing valuation work. And this leads me to the final thought, you may need to steer the conversation away from ROI calculations in the short term. Like, just even having this conversation may not be really what you need to be focused on.

Why? Well, for years, companies and orgs have invested in things that don’t always have a short term ROI. They know that it takes time, and they can’t really measure it along the way what it’s worth. Examples include investing in product dev, materials, or resources in order to create a premium product. Apple’s the classic example of this. Should we spend all this extra money to develop the machines just so we can have rounded corners on these products? And our screws, the edge of the screws also needs to be beveled and rounded off so it’s a completely smooth back case. What’s that worth?

Companies like this understand that, like, that’s almost the wrong question to have because it’s certainly going to be a fraction of what the price premium is that we’ll be able to charge in the future that’s associated with our brand as a premium product. Another example: ESG and ethics. Because you know your customers want to buy from respectable companies, and you, or your stakeholders, want to be known by the market and regulators as being an ethically-minded company, so they invest in things like ESG and ethics. Brand advertising. I mean, this is one of the things a ton of money is spent on brand advertising where there’s not even a call to action about buying the product, necessarily.

And is it worth it? Well, when a PR disaster strikes, your brand might be all you have, and all of a sudden, all that money that was put into the brand becomes super important, and the value is very clear. But along the way, is there an immediate ROI? Probably not. Culture and employee engagement. R&D labs and experiments.

I mean, look at—like, my understanding is deep learning and neural nets and work at Google, a lot of this stuff. Like these were R&D projects that did not have a specific commercial point or purpose at the time, but those became foundations. Innovation work. I mean, hello, product people, right? Look at the most senior levels, most leaders know you can’t just keep repeating what you did yesterday and expect nothing in the market to change, and you’re going to be just fine.

If you’re just repeating yesterday over and over again, you’re eventually going to be dead because people change, buyers change, the market changes, global economy. Most of you listening to this show are smart enough to know this. So, some smart companies are doing innovation work, unless they’re totally checked out, and they’re just waiting for retirement, and they don’t want to change anything. And I’m sure we all know examples of this where your management or your leaders are not doing this work because they probably have a short-term mindset about something. But at the highest levels of the organization, usually there’s people that do care about this because they understand we can’t stand still.

And so, in the short term, you invest in innovation not knowing what the exact ROI is going to be. Is this hard to do? Yes, but it’s easier if you have built up trust and helped your stakeholders and customers achieve their, frankly, previously unstated qualitative goals. If you’re constantly playing defense and having to justify your existence by quantifying the value of your data products, or your data product management team, or UX team, and you’re working in hard numbers all the time, then that may just simply mean the value of your work is still too opaque for stakeholders to see, and they’re looking for a hard number to assign to it because they’re not seeing the other forms of value that they care about. It’s easier to write you a large check for a big, innovative, unproven initiative if your stakeholders know what you and your team can do with a small check. Again, if you want a larger investment, you’re like, “Hey, I need to plant the seeds for this. We’re not going to get vegetables the first year. We got to plant some seeds and we got to till the soil first,” it’s easier for a stakeholder to write that check for you, if you’ve proven what you can do with a small check.

So, that’s basically it. A lot of these concepts, again, they don’t apply just to data products, and these can be used for all kinds of situations where we need to estimate value, but I’m hoping this will help you to learn how to both measure value when it’s necessary and to change the conversation, to focus on what’s really most important when short term ROI really isn’t, and to estimate value when contributions are indirect. I might do a workshop on this topic, and I really tend to listen to feedback from my email list, which you can hop on. I send out every Tuesday, designingforanalytics.com/list, so shoot me a message there, just hit reply, or you can email me, brian@designingforanalytics.com.

I’m kind of curious to see if this is something that people would like some hands-on help with. I might run a little trial workshop or something like that, and consider doing an offering around this. So, let me know. If I don’t hear then I’ll assume that you have this under control, and if I do, then we’ll take it from there. So also, I just wanted to thank all the listeners who have provided reviews for this show. I’m shocked that this show is in the top 2% of podcasts in the world. I really did not set out to become a podcaster, and I don’t even really use that title. It’s just my way of sharing my insights with you and hoping to be helpful to this community.

So, thank you for your listens, thank you for your reviews. If you haven’t left a review, you can do that fairly easily in Apple Podcast. That’s where it makes the most bang for the buck in terms of helping get the show out there. If you go to designingforanalytics.com on the podcast page, there is actually a way to visually see how to leave a review, just in, like, five minutes. I have a little visual guide there. So, check out designingforanalytics.com/ed for Experiencing Data, and you’ll see a link there to how to leave a review. So again, thank you, and we’ll see you on the next episode.