Episode Description

In today’s episode, I’m joined by John Felushko, a product manager at LabStats who impressed me after we recently had a 1x1 call together. John and his team have developed a successful product that helps universities track and optimize their software and hardware usage so schools make smart investments. However, John also shares how culture and value are very tied together—and why their product isn’t a fit for every school, and every country. John shares how important  customer relationships are , how his team designs great analytics user experiences, how they do user research, and what he learned making high-end winter sports products that’s relevant to leading a SAAS analytics product. Combined with John’s background in history and the political economy of finance, John paints some very colorful stories about what they’re getting right—and how they’ve course corrected over the years at LabStats.

Highlights/ Skip to:

• (0:46) What is the LabStats product
• (2:59) Orienting analytics around customer value instead of IT/data
• (5:51) "Producer of Persistently Profitable Product Process"
• (11:22) How they make product adjustments based on previous failures
• (15:55) Why a lack of cultural understanding caused LabStats to fail internationally
• (18:43) Quantifying value beyond dollars and cents
• (25:23) How John is able to work so closely with his customers without barriers
• (30:24) Who makes up the LabStats product research team
• (35:04) ​​How strong customer relationships help inform the UX design process
• (38:29) Getting senior management to accept that you can't regularly and accurately predict when you’ll be feature-complete and ship
• (43:51) Where John learned his skills as a successful product manager
• (47:20) Where you can go to cultivate the non-technical skills to help you become a better SAAS analytics product leader
• (51:00) What advice would John Felushko have given himself 10 years ago?
• (56:19) Where you can find more from John Felushko

Quotes from Today’s Episode

• “The product process is [essentially] really nothing more than the scientific method applied to business. Every product is an experiment - it has a hypothesis about a problem it solves. At LabStats [we have a process] where we go out and clearly articulate the problem. We clearly identify who the customers are, and who are [people at other colleges] having that problem. Incrementally and as inexpensively as possible, [we] test our solutions against those specific customers. The success rate [of testing solutions by cross-referencing with other customers] has been extremely high.” - John Felushko (6:46)
• “One of the failures I see in Americans is that we don’t realize how much culture matters. Americans have this bias to believe that whatever is valuable in my culture is valuable in other cultures. Value is entirely culturally determined and subjective. Value isn’t a number on a spreadsheet. [LabStats positioned our producty] as something that helps you save money and be financially efficient. In French government culture, financial efficiency is not a top priority. Spending government money on things like education is seen as a positive good. The more money you can spend on it, the better.  So, the whole message of financial efficiency wasn’t going to work in that market.” - John Felushko (16:35)
• “What I’m really selling with data products is confidence. I’m selling assurance. I’m selling an emotion. Before I was a product manager, I spent about ten years in outdoor retail, selling backpacks and boots. What I learned from that is you’re always selling emotion, at every level. If you can articulate the ROI, the real value is that the buyer has confidence they bought the right thing.” - John Felushko (20:29)
• “[LabStats] has three massive, multi-million dollar horror stories in our past where we [spent] millions of dollars in development work for no results. No ROI. Horror stories are what shape people’s values more than anything else. Avoiding negative outcomes is what people avoid more than anything else. [It’s important to] tell those stories and perpetuate those [lessons] through the culture of your organization. These are the times we screwed up, and this is what we learned from it—do you want to screw up like that again because we learned not to do that.” - John Felushko (38:45)
• “There’s an old description of a product manager, like, ‘Oh, they come across as the smartest person in the room.’ Well, how do you become that person? Expand your view, and expand the amount of information you consume as widely as possible. That’s so important to UX design and thinking about what went wrong. Why are some customers super happy and some customers not? What is the difference between those two groups of people? Is it culture? Is it time? Is it mental ability? Is it the size of the screen they’re looking at my product on? What variables can I define and rule out, and what data sources do I have to answer all those questions? It’s just the normal product manager thing—constant curiosity.” -John Felushko (48:04)
  

Links

• John on LinkedIn

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today I have John Felushko on the line from LabStats. John, welcome.

John: Great to be here.

Brian: Yeah, LabStats, it sounds like science software.

John: It is not.

Brian: That is not what we’re talking about [laugh]. We’re not talking about doing anything in a lab here, right? So, maybe I will tell you, and then you can tell me if I got it right because this might be a good test of your positioning. And effectively, from our chat, like, a week or two ago that we had, I was like, “I didn’t even know this existed, but it makes total sense to me.” LabStats is software that is installed in academic settings to try to track usage of both software and hardware, so computers and the software that’s been installed on them, so that better purchasing decisions can be made and resources can be allocated, and we’re not wasting money buying software students and faculty aren’t using. How did I do? Ten out of ten? Nine out of ten?

John: Yeah, that’s nine out of ten. Nine-and-a-half out of ten.

Brian: Oh, wow. Excellent. What’s the half I missed? Tell us about that.

John: So, the really cool thing that we can do is we can tie that usage of those resources back to the students, and then help people understand the usage of those resources in terms of the outcomes for the students. So, things like, oh, how does this impact their GPA? How does this impact their graduation rate? So, it’s not just dollars, but also outcomes.

Brian: Oh, I see. That almost seems like a different user group would care about that, whereas IT is, like, looking at, like, someone needs a Mac 15-inch quad processor, blah, blah, blah, and I don’t know what they’re going to use it for, but here it is, with this package on it. Like that’s a different audience than the academic one. Tell me about that.

John: It was a different audience. Universities are pretty stove-piped as customers, so their different values are structured in different organizations, but there’s a massive demographic cliff hitting universities, and they’re having to think about how to restructure their organizations in really fundamental ways, and they’re starting to think about IT as no longer a service provider, as much as IT is a core enabler of their core functions, and we’re providing the data that helps them become that. So, we’re really at a paradigm transformation point for our customers.

Brian: I’m kind of surprised about what you just said because I feel like in the enterprise data product management crowd that I talk to that’s, like, a dream for what IT is supposed to be, but data is often associated with the IT department, and they’re really still a service organization with—you know, I know people listening to this show are often trying to be the ones changing this to be focused on delivering benefits and outcomes. But you’re telling me that academia is actually already, like, doing this more actively, and that just surprised me that they’re connecting the dots, that IT is not just a—we provide oxygen and clean air and water and, like, a facility, but rather, it has something very directly tied to the goals of the universities and that academic setting. Am I off on that? Or, like—

John: Yeah, it’s a new idea for universities. It’s not very widespread.

Brian: Oh, okay.

John: I’d say it’s under, say, 15% of the orgs I talk to are actually doing that, but everybody wants to do that at the CIO and analyst level. So, you know, we’re selling into an emerging need, and a need that people, because of the demographic, the underlying economics of their situation, most people realize it’s change-or-die at most of these organizations. And so, the pressure is there.

Brian: Is this rooted in kind of the squishing of the middle university market that I keep hearing about, basically? You know, colleges are closing, and if you’re not at the extreme—if you’re at the budget end way down at the bottom, or you’re at Harvard, the middle is getting squashed quite a bit now that coursework can be delivered online, and you don’t need to necessarily travel to get somewhere, unless you want to buy the [laugh] brand, so to speak. But I keep hearing the middle of the university ecosystem is most at risk.

John: Yeah, it’s an underlying—this is one of the things I think is super important for product people to understand, is understand the big picture forces shaping the market that they’re selling it to. And so, we just had news out of China in the last couple of weeks that they’ve been overcounting their under-40 population by a couple of hundred million people. We’ve also known from US demographics, that the birth rate since 2008 has fallen by 30%. You combine those two factors, the number of people available to enter university—number of high school graduates, essentially, every year—is dropping.

And then the boomers are all retiring; there’s massive shortages in blue collar jobs. Well, that’s going to pull people out of the university stream. So, between all those things, yeah, the middle of the university market is collapsing. Community colleges are going high, and then the people who sell Veblen goods—so goods that become more valuable as they become more expensive—like Harvard, they’re probably going to be fine. But every state, local college in the US and their equivalents around the world is under enormous pressure.

Brian: Yeah. The reason I had you on the show is I was really impressed with our conversation that we had, and I was like, this is someone that’s doing solid bread-and-butter product work the way that I think it needs to be done. It seems like you’ve got a process that you’re following. And I noticed your LinkedIn tagline, do you want to say it—Producer of Persistently Profitable Product Process, and that’s 5p there.

John: Yeah.

Brian: Is that something that you formulate, that you actually have a formal thing you’d like to talk about? But it sounds like you’ve had some real successes building this product with your team and turning this into, like, a profitable company, especially in a market that’s struggling right now, but they’re seeing your service as essential, it sounds like.

John: Yeah, so the product process, that’s the key to me, is… you have people trust the process, which is really nothing more than the scientific method applied to business. Every product is an experiment, has a hypothesis, you know, a hypothesis about a problem it solves, or really a nested set of overlapping problems: a customer’s problems, and the business’s problems. And so, yeah, that process is something I’ve developed in my 16 years in product management. Had a lot of failure and analyzing, really in-depth, my own failures in developing. Why did this product fail?

And so, we really have a process at LabStats where we go out, and we very clearly articulate the problem. We clearly identify who the customers are, down to the point where my requirements docs list specific customers by name, who are examples of people having that problem, or prospects by name. No, no, no, not personas. John Smith at this college has this problem. Here’s the example. And then we really incrementally, as inexpensively as possible, test our solutions against those specific customers, and we find that almost always, if we’ve done a good job of picking our customers and qualifying them, the success rate’s been extremely high.

Brian: How do you manage the fact that John Smith’s way of doing it isn’t the way Jane Doe does it over on the East Coast? I hear frequently, “Oh, everyone needs a customized view the dashboard,” et cetera, et cetera, et cetera. How do you mitigate that as a product manager? And, I mean, that’s a messy—that’s a terrible question because it’s like, there’s so many, ‘it depends’ in there, but do you have any general [laugh]—

John: What I like to do to deal with that diversity? I like to provide people with an à la carte. What we’re doing in data is providing people with really powerful à la carte solutions for those people who want it customized, and realizing that they’re going to be the loud users, they’re going to be the power users, but they’re going to be the minority of my users and the minority of my revenue, and then, developing super simple, dumb… dumber solutions, less customizable solutions that account for the majority. And we’re constantly asking about every problem and every request. Hey, is this problem persistent? Is it profitable? Is it painful?

A lot of the things people ask for, a lot of the features people ask for, when you ask the problem behind that feature persistent, profitable, and painful, the answer is no. And then you’re like, okay, here’s our à la carte solution. In our case, what we did is we built APIs, and we built Power BI templates that can be customized. And I’ll spend an hour or two, or my data scientist will spend an hour or two, building a customized template for that one-off customer. Doesn’t have to be fully tested, it doesn’t have to be fully fleshed out. It solves their problem, and then we can go back to focusing most of our effort on building really smart solutions that answer the majority of the very common needs.

Brian: So, just to help people visualize this, my understanding is, LabStats is the bread and butter version for the non-power user, is delivered in a web browser as a standard web application, probably hosted in the cloud, maybe not.

John: Yeah.

Brian: And then you have this, like, the API and Power BI integration thing, where they roll their own views, or you roll them for them to get them started, or whatever it may be, but something along those lines. Is that basically correct?

John: Yeah, that’s exactly it. And so, we have customers that never log into our web portal, that only interact with our data through our APIs, customers who pretty much only interact with the product through the Power BI templates we build and the customized dashboards they’ve built on top of those, and then we have the majority of customers that do the web product. So, it’s really important to satisfy both groups because the customer base is so diverse. You have customers, a typical, like, a big Australian school. Big Australian schools—so in Australia, 1 in every 11 people is a foreign student. Tuition is the third-largest source of GDP for Australia, foreign student tuition is their third-largest source of GDP.

Brian: Wow.

John: So, their schools function like huge, sophisticated, profit-producing businesses because that’s what they are. And so, they have a completely different set of needs than a Nebraska Community College with 500 people enrolled, and a whole different set of analytic capacity. Like, analytic capacity, what we call Type 2, or System 2 thinking capacity between these organizations is really, really diverse, and so we have to kind of constantly asking ourselves, what’s the cheapest way to satisfy both customer bases.

Brian: Right. You mentioned following the process, I sometimes talk and correct me if I’m—not how you model it, but I think of that as, like, product as the verb and not the noun. It’s the way we do the work, right? And you said that you’ve changed that. And you said, you know, there’s some failures in there.

Is there an example of something where your experiment failed, or a learning that you had, or an assumption that you made, something maybe recent, that you could talk about, that someone just to help someone make that concrete about what does it mean when you talk about failing and adjusting your process or methods?

John: So, when I started at LabStats, we decided we were going to—this was under different ownership—I walked in, and every time I walked in as a product manager, there’s always some big initiative underway already that you’re just dropped into the middle of—

Brian: Right, right.

John: And the big initiative was we were going to internationalize. We were going to go out and go after all of these markets, all around the world. And the very first thing I asked was, “Has anybody ever talked to these markets? How many French customers have we talked to? How many Latin American customers have we talked to? How many Indian customers have we talked to?” Prospects. The answer was zero.

But the leadership team at the time was completely sold on that concept. This was going to work. Nobody talked to them. We spent all the time and money internationalizing it. It completely failed. We got no customers. So, that’s a really broad example of not following the first process, which is, don’t do anything until you can assign specific customers to that problem. We know that they have the problem, you know, it can be sold, we know they’d be happy with the product. And that’s a pretty simple example.

And then I’ve had stuff fail all the way on the end. It was a line of hard goods, and it had a very specific set of purposes, and I happened to leave that company just before launch marketing was complete. And I got a chance to go back and run into their salespeople who didn’t know me, right after launch. And so, I pulled up the product I’d worked on for two years, and was like, “Tell me about this. Tell me why it’s valuable. Tell me why I’d want to buy it.” And the sales and marketing team could simply not articulate about 90% of the cost of that product.

So, a product could fail at any point along that process, right? At any point along the continuum. But they’re always kind of really simple, they’re always a disconnect. Like, you did not have a painful, persistent, profitable to solve a problem, or you did not articulate the value of your solution to that problem. All the way along, it’s about attaching it to value, right?

And that value, like, you can flip that and say you didn’t articulate the value to the business, where you built a product that wasn’t profitable, well, doesn’t value the business, or you built a product that’s too risky, that’s not valuable to business. Really important to be like, okay, what’s the problem we’re trying to solve for the customer, and what’s the problem we’re trying to solve for the business, and keeping both of those equally weighted throughout the entire process, and then telling people about them. It literally doesn’t matter what you build if you can’t communicate it to the customers and the stakeholders. That’s probably the biggest consistent failure I see in product is people not communicating to people who benefit, how they benefit. I was in a stand-up this morning where we launched a solution that would make sales life way easier, and we made decisions today that we did not articulate to them, and they’re upset about it.

Brian: Why were they upset?

John: Because it didn’t work the way that they thought it would work in their head. It didn’t work—and we made product—we’re like, and we didn’t take the time as we’re doing it, to articulate it to them. “We know this is what you asked for. This is what we think is better. This is why we did it.” And we weren’t doing the work of doing the communication. Engineering was like, “That’s just going to slow us down.” It’s like, “No. [laugh]. No, no, no.” Like, “Yes, it will slow you down, engineering, to talk to customers, talk to users—” In this case, it was internal users, “—but if they don’t believe in it, there’s no point in doing it.”

Brian: Yeah, that whenever I hear the slowness, it’s like, yes, the outputs will come out slower now. The outcomes might come faster because we might have to build it two times now, so the outputs will roll out slower. And everyone wants to wait for the outputs, but that’s really the beginning of the game; it’s not the end of the game [laugh]—

John: Right.

Brian: —you know? But you know that. I’m curious, just a little tangent here, why wouldn’t this work in the French university system? Why wouldn’t it work in Switzerland? Why wouldn’t it work in India? They can’t be that different. Universities have been around for hundreds of years. Internationalize the product, get it done. It can’t be that hard. Why do you think that failed? And you said they didn’t talk to the customers, but clearly there’s that that can be one of those, like, baffling, how could this not have worked. What happened?

John: Yeah, all the people, when I got here, this company was very local in Eastern Idaho. All the people were from Eastern Idaho. They’d had one culture. I grew up in America and Canada, and I’ve kind of traveled around the world pretty extensively. One of the failures I see, especially in Americans, is we don’t realize how much culture matters.

Americans have this bias to believe that whatever is valuable in my culture is valuable in other cultures. And especially when your customers are large organizations, or their governments, or their NGOs or… value is entirely culturally determined. Value is entirely subjective, right? Value isn’t a number on a spreadsheet, in almost all cases. And because it’s subjective, it’s therefore determined by culture.

And so, the product we had at the time and the way it was being sold as something that helps you save money and be financially efficient. Well, in French government culture, financial efficiency is not a top priority. It’s just not. Spending money, spending government money on things like education is seen as a positive good. The more money you can spend on it, the better. So, the whole message of financial efficiency wasn’t going to work in that market. Nobody talked to the customers and asked why five times, got to the bottom of that.

And I see that over and over and over again in my product career. I’ve done it too. I built something I thought was awesome, I thought looked cool, that fit in my esthetic, even. Totally didn’t work with the culture I was trying to sell it into. And so, culture is a real thing, diversity is a real thing. And I know everybody says it, and we all talk about multiculturalism, and we all talk about, like, oh, it’s a wide world, and diversity, equity, inclusion is important, but I don’t see people, like, actually thinking about what that means very much in the product process and questioning their own biases.

And the biases at a really fundamental level, like ‘saving money is valuable.’ ‘Saving money is valuable, saving money is good,’ isn’t true everywhere, in every situation, right? And it’s really difficult when you—especially when you have a company that’s largely drawn from one culture and a group of people that are largely drawn from one culture, to get people to question those fundamental assumptions.

Brian: Yeah. You talked about all value is subjective, so I just wrote something to the mailing list about this. And this is particularly true, I feel like, in the enterprise data product management world. So, the listeners that are focused on their value comes from internal decision support and other forms of help using analytics and machine learning and AI. They’re not selling their wares as software.

So, they are so relentlessly focused on quantifying in monetary units what the value of their work is, which is often buried inside a larger stack. So, it might be the intelligence portion of an application that someone in sales uses in order to do pricing for some other thing, and it’s like buried inside there, and all the talk is about the financial dollar benefit. Which does matter, and I think good product people know that. I mean, this is ABCs; you need to be able to show ROI. But since value is subjective and culture can different, can you unpack a little bit about the other types of value?

And I had talked about particularly, like, your software for example. Like if you made the leadership team at the university feel like they’re on top of the world, they know what’s going on, they’re in control, they’re making good investments, yeah, there’s a dollar thing, but is their empowerment—their feeling of empowerment, is there value in that? It’s harder to quantify that in dollars, but that might be the reason they renew, is the feeling that they get. And the loss of control if they didn’t renew, they would feel like we are flying completely blind if we don’t have this data. We are flying blind. And it’s like insurance. They’re buying insurance. Can you talk about this, like, non-monetary—or maybe you don’t think of it that way. I don’t know. Do you think of everything back to dollars? Or—

John: Yeah, yeah, that’s exactly how we think about it. So, like, what I’m really selling with more data product is confidence, right? I’m selling assurance. I’m selling an emotion. Before I was a product manager, I spent about ten years in outdoor retail, selling backpacks and boots.

And what I learned from that is like, what you’re always selling is emotion, at every level. And yeah, the ROI is nice, but the ROI really just leads—if you can articulate the ROI, the real value is that the buyer has confidence they bought the right thing. The ROI is just a step, right? So, we have this journey-to-value timeline mapped out where we’re like, “Okay, here’s our customer’s journey through our product to whatever problem they’re trying—” and the product it ends with, “We have realized value.” Not our product has delivered value, but our user and our customer believes we have delivered value. It’s a very, very, very different thing.

And it’s easy to get confused [laugh]. It’s really easy to get confused and say, “Hey, look—” And I routinely have calls with customers where I’ll pull up their data, and I’ll be like, “Look, our product found you $4 million in savings.” And they’re like, “I can’t ever realize that because I don’t have the staff to realize that,” or, “Saving money isn’t a priority for us right now,” or all kinds of different reasons why they can’t get to realized value. And so, when I’m doing product [unintelligible 00:21:54], I’m always like, “Okay, how does this help the customer get closer to them realizing value, them believing subjectively that they’ve achieved value?”

And the only way I found to figure that out is spend a ton of time talking to customers. So, we record every sales call, and product and my engineering manager, we listen to every one of those calls. We record most support conversations, and we listen to a pretty good conversation of those. I have a standing rule for support or sales that if they need somebody at any time, day or night, anywhere in the world, they need someone to jump on, ping me, and I’ll jump on the call with them. So, I’m constantly available to be there at the point of pain, right? It’s way more valuable to talk to somebody who’s specifically experiencing pain or confusion, than to talk to them outside of that context.

And then we do tons of surveys. I go to every trade show. When I go to a trade show and I’m with customers, I don’t ever eat a meal alone [laugh]. I’m talking to customers from breakfast till midnight every day. And it’s just, like, immersing yourself in their world and understanding their values and constantly asking questions like, “Hey, what are your top priorities? What would cause you to lose your job? What would make you a hero to your boss? How would you get promoted two steps?” Right?

And just understanding, like, what matters to the user, and can then can we put it—because what we found is we can deliver the data, but if we don’t have the right language around it, they don’t appreciate the value of that language. So, that paper value, those numbers on the spreadsheet or that graph doesn’t become value till it’s explained in English, in the right words to them.

Brian: Do you have an example of that where you change something, or you, maybe you realigned it, or anything like that.

John: Dumb stuff, like, really basic stuff. We used to have a data type called logins that counted the number of times users logged into a computer or a piece of software. We changed that name to users. All we did was change the name of the data type: what we’re counting as users, user interactions instead of logins. And people went, “Oh, that’s way more valuable.”

Brian: Even though a user has multiple logins?

John: We changed it, and we changed the perspective to users. We started calling it users. We’re talking about users. We’re counting user logins, right, but we need to talk about specific logins, but we changed the main architecture of the page so that was in a context of users—

Brian: Okay.

John: —right? It’s a context of people.

Brian: Oh, I see. It was too granular having it down because, like, oh, one person had 20% of the logins. Like, some bot was, like, going and—

John: Right, and then even when you look at a report now for logins, we call it ‘user logins,’ and not just logins, right? So, the context is, oh, this is people.

Brian: Understood. Yeah.

John: Right?

Brian: Yeah. Their mental model of the world is that users is a logical object that is—that’s something that I should be tracking.

John: Well, and users are what they care about.

Brian: Right.

John: Right? What’s a login? Who’s a login? What—who cares about a login? Users are things they’re emotionally connected with. Most of my customers, in interviewing them, came up from the support desk. They started at the support desk at the university, and then they moved up into higher management. So, a user is a thing with a face with emotional resonance to them. So, we’re always looking for how do we frame this so it has better emotional resonance? How do we frame this so it’s a more immediately recognizable? How do we frame this so it’s easier to interpret?

Brian: Another reason I wanted to have you on the show was like, I mean, I don’t remember the last time I talked to someone in product that seems to have as much customer exposure hours as you do. I mean, you talked about this extensively, about the amount of time you have with them, but also, the thing that stuck out to me was the level of access that you seem to have. I would call them, like, design partners, but when end customers are willing to get on a call, or give you feedback over email on a dashboard, or whatever the heck it is, did you just luck into a very verbal customer audience? Like, at so many places, it’s either hard to get access to them, or they say we don’t have time, or there’s a blocker, you know, someone’s standing—the sales is standing in the way. Did you grow this? Tell me about how you’ve managed to get this access and to prioritize that time in your own schedule.

John: So, we went and said, “Okay, who has motivation to talk to us?” So, we created this thing, we call it an influencer program, and the initial group of people I reached out to for the influencer program who are the people I usually ask questions to are anybody who has a support ticket or a complaint. And we immediately, as product, typically, we’re reaching out to them and saying, “Hey, we know you’ve got problems with our product. We want to talk to you about how we can make it better for you.” So, in our customer communication, first of all, we try and talk to—we talk to everybody; we’ll offer everybody that opportunity, but we really focus on the people with pain points because they’re more willing to talk to us. And then we always frame our message in, “Don’t help us. Help us help you. Help us solve—” so every communication, the very first sentence is, “This is how it’s going to help you.”

Brian: Yeah.

John: “This is about you as the user. Let us make your life better, your user experience better.” And then I go to—a lot of the contact and relationships has gone from going and speaking at professional conferences, and going and saying, “Hey, I have some experiments that I’ve seen people try to make their lives better, people like you, whose experiments I’ve seen. I want to talk to you about the experiments I think you can run to make your life better. And here’s a little bit about how our product can help you.”

But more, it’s about paradigms, concept, thought leadership, but being, like, we always frame it as super humble. Like, hey, I really want to know about you and your pain point and your value and your context. And those speaking engagements have turned into great relationships with our customers. Pretty decent sales, too [laugh], could be traced back to those, but great relationships with the customers. And then we are constantly doing webinars that are the same context.

So, we’re constantly just giving people an [exposure 00:28:03] to talk to us. In fact, in our product, we just added a picture of all of us in the support team, and myself, our photograph on the main page of the product with a ‘Contact Us’ or ‘set up a meeting with us’ button.

Brian: Not the faceless LabStats.

John: Yeah. Like, you’re humans. Me, John. I want to talk to you, and I want to help your life be better. And that just human… that human connection, you know? I think of my—like, the persona I like to adopt is, like, the best waiter or the best shoe salesman you’ve ever met in your life, right? That person who, like, genuinely cares about getting you the best outcome. I do genuinely care, and then I’m like, how do I show that I genuinely care?

Brian: Yeah. Yeah. If you’re listening right now—well, you are listening because you wouldn’t hear me ask if you’re listening; that’s like, did the tree make a sound when it fell in the forest? Man, that was terrible. Maybe we should delete this part [laugh]—but what I wanted to say was, we’re talking about product, we’re talking about data products, and analytics, and insights products, but we’re so much not really talking about the product. We’re talking about the benefits.

John: Yes.

Brian: We’re talking about the value as perceived by them and not us. And us can talk about the dollars part, which is, as you said, that’s actually a byproduct of the value being created, right? It’s the money then falls in at the end because the purchase happened, and they assume they’re going to get value by giving that credit card over, and then the value comes. The money comes later, actually [laugh]. It’s a byproduct of the value of the benefit.

John: I really like the whole Amazon method, where you start with the—the very first thing you write down, you have a new product idea, is the press release.

Brian: Yeah, yeah.

John: Okay? And then I say, “Okay, here’s the press release. Who would this go out to? Who would it benefit?” And then, very often, the first—I was doing this with a product concept yesterday—when I have a new concept, often the first people I reach out to are my sales team, and I say, “Okay, if I gave you this pitch, could you sell it?” “Yeah.” [laugh]. “Okay, cool.”

And then the then it comes back to, can we make money on this? Can we find a profitable way to deliver those benefits? But it’s very much like the customers have a real problem, we’re embedded in their world, we feel their problems, we empathize with their problems, and then you’re doing the magical thing of, like, trying to match up the technology you know is available to create a solution for that.

Brian: Yeah. I think you said that you have some user experience professionals within your product team. I’m curious, are they part of this research effort? Are they researchers or designers? Are they part of this or do you tend to do that work, and then you’re passing back—

John: No, I—

Brian: —in direction to them? Like, how does that work?

John: Almost every conversation I have with a customer, there’s an engineer, a UX designer, or a salesperson with me. And I think the whole—or every single person in the whole organization—in fact, when we got a new CEO a few years back, the very first thing I asked him was like, “Let’s get on some customer calls.” “Do you want to tell me about the product?” I’m like, “No. First, I want you to experience the product as a customer would, and then as soon as you’ve done that, you and me, we’re going to go do customer calls. You and me, we’re going to go get on airplanes and go talk to customers.”

And everybody through the whole org has to be involved in the research process. We have an Intel channel in Slack where everybody posts everything they learned. Like, just tell me what’s going on. Tell me about your customer conversations. And every time somebody has a product idea—which I know, product managers are constantly played for, everybody thinks they know where the product should be—you’re like, “Awesome. Thanks for that idea. What customer did that come from? What problem were they experiencing them? Can I talk to them? Let’s talk to them together.”

Because so many of the conflicts I’ve seen in organizations are because we don’t have a common operating picture. We don’t have a common set of beliefs about who the customer is and the problems they’re facing. And so yeah, research is never, never, never a one-on-one [laugh] activity. It’s always a group activity. Everybody talks to them.

And then as soon as UX has a design, it’s like, great, let’s go call some customers and show them this sketch and see what they think. Let’s talk to support and run this through them, and be like, hey, support, what here that we just designed was going to generate support calls? Why? Right? That whole get over, get all the players on board, get the value out of everybody’s brains. Research is a team sport. In fact, I kind of don’t like the idea of researchers in companies, unless they’re very, very large because it takes the onus off everybody doing research all the time.

Brian: The way I like to think of it is, like, you have an expert that’s helping to democratize the research activity, to make sure we’re doing quality. We’re improving the quality of how we do the research, but we’re not taking other people out of it. We want to evangelize that work. It’s the same thing with design. I think design can be done by non-designers, but if you don’t know what you don’t know about it, it’s hard to know if you’re doing it well.

And everybody can look at something and have a personal opinion about it, so everybody has an opinion about design, you know? And so, how do we get better at those things that when we don’t know… we don’t know what better looks like, or we don’t know. And that’s the purpose of having expertise in-house. It was great to hear that you are evangelizing that process. I’m not sure how much that’s happening in the data science and analytics space.

I feel like there’s still a lot of focus on data. People are doing model development, and ETL and building pipelines, and it’s still so much about the plumbing, and then there’s this one or two people that have access to the end-users. You’re getting that one perspective, and that person may have come from a data lineage. They might have been an analyst or a data scientist themselves, and they decide they’re going to go into product.

And it’s a start. It’s going from that zero to one, but I’m with you that having the perspective of an engineer to be able to say that’s really great, but there’s privacy considerations, or getting this data in real time, that’s in batch, or this is coming in batches, and this is not, so now the number of users is not going to line up because X, Y and Z. So, great idea, but it’s always going to lie. And getting those different perspectives is so valuable because we all bring a different wheelbarrow of stuff from our experience and [laugh] and what we carry along with us. So yeah, I love that [laugh].

John: I’m a big fan of whenever—like, as soon as it’s about the customer, let’s get off Slack, let’s get off email, let’s get on calls, let’s do it face-to-face, and then, oh, this touches on the data scientists area. Hey, data scientists, jump on this call. And we just have this culture now where everybody’s like, you drop everything when there’s an opportunity to talk to a customer. Nothing is more important across the whole organization than talking to a customer. And let’s just expect it.

And it works really well. And it makes design, especially when we’re in a situation where you have to solve something really fast, you don’t then necessarily have to go out and do new research because everybody has so much of the customer’s experience embedded in them, we can go out and do an MVP super quick, sometimes in hours because we were all constantly talking to users.

Brian: So, you get all this feedback, you’re doing this customer exposure time, you’re getting these insights there. How and when does design get involved with that? Is it getting routed through you and then to them if they were not on the two-on-one, you know, research call that you had? When do you need design versus do it yourself? I’m a PM; I think a lot of the data product managers out there are de facto doing design work because if they’re new to product, then they’re getting started, and they don’t have that leg of the triad yet. And so, they’re effectively doing it themselves, whether they realize they are or not. I’m curious, when do you know you need a designer? When do you work with them? How do you work with them, especially when you have these insights from customers?

John: Yeah. So, I know that the way I read and the way I look at data is deeply not typical. I’m not going to see it, I can’t see it the way most of our users would see it. My brain doesn’t work like that. And so, I always have a designer, or a copy editor, or—I always have somebody else review everything I do. And now that we have a designer—in the last two years, we’ve had a designer—designer touches everything.

And I really work hard, what I’ve learned working with designers—and I came from working at a company called Arc’teryx, originally, was my first product job—and it was physical goods. And I found that designers got really prickly when I told them what it needed to look like, when I told them how it needed to feel. It’s like, “No, no, here’s the problem. Solve it, or better yet, hey, designer, here’s the customer with the problem.” But I really, really try and stay out of specifying design as much as I possibly can.

It’s not my strength because I’m so atypical in how I read data and how I read, and so knowing myself, I don’t ever do a design work myself. Because what I will design is something that works for me, and I’m such an outlier that it wouldn’t work for anybody else. It’s like I have a giant head. When I designed hats, if the hat fit me, it wasn’t going to sell well.

Brian: [laugh].

John: Right? I knew that about myself, right? And I’m like, you know, because one of the things you do when you’re designing physical goods is they’re like, hey, they bring it to you, and it’s a clothing, they have you put it on. And I’m like, “Hey man, if, like, if the hat fits me, we can’t sell this.” I’m point-zero-five percent of the population.

If the copy makes sense to me, it doesn’t make sense. It’s going to need—and so it’s like, just pulling myself out of the equation as much as possible. And I’m way, way, way too embedded in the product, in the meaning and the raw data to be able to be a good designer, right? I’m so different from my customer because I do this all day, every day, that I’ve automatically biased myself to the point where I’m useless at reviewing the design work.

Brian: Yeah. Tell me about your evaluation method, then, to—you get artifacts back.

John: I get customers, or, like, as close as I can get to customers in the time I have to review it for me. “You tell me what you see.”

Brian: “John, we don’t have time for that. We don’t have time for that, John.”

John: [laugh]. Do you have time to do it twice?

Brian: [laugh]. No.

John: Three times? Four times? 19 times? Do you have time to be wrong?

Brian: Yeah.

John: Right?

Brian: There’s always budget to build it twice [laugh].

John: That stuff drives me up the—when people give me a ship date. Like, we have a dateless culture here. We don’t use dates ever. Stuff ships when it’s done, when it’s good enough, and we have high confidence that it’s good enough.

Brian: How did you get to that? That’s a huge statement you just made. I mean, this gets back to the original Agile Manifesto is accepting that you can’t predict finished dates and all this kind of stuff, but getting the org—especially business, non-product business people—to accept that is extremely hard.

John: We have three massive, multi-million dollar horror stories in our past where we didn’t do that, and it cost us millions of dollars in development work for no results, no ROI. So, every time that comes up, I’m like, “Hey, do you want to repeat that process again? That’s what we did here and here and here.” Horror stories are what shapes people’s value more than anything else. Avoiding negative outcomes—a specific negative outcome—is what people avoid more than anything else.

When I first started out in making hard goods, I made stuff for commandos, and so I meet with commando organizations all around the world. Can’t really get into their names, but like… and some people would have some really—one group was like, “Does it melt when you light it on fire?” And I’m like, “Yes.” They’re like, “We can’t buy it.” “Why?”

Twenty years before, they’d had a very public event where they went into a burning building to rescue some people, and their guys got caught on fire, and avoiding that negative outcome is their frame for value, 20, 30, years later. And so, knowing those stories, and telling those stories, and perpetuating those stories through the culture of your organization—these are the times we screwed up, and this is what we learned from it—do you want to screw up like that again because we learned not to do that.

We have a book club that’s most of the company. It’s, we’re always sharing books [laugh], and so that’s more common frame of, like, “Hey, remember this from this book?” “Oh, yeah.” “Do you want to do that?” “No.” When people want to override product, I make them do it in writing. Like, “Hey, you’re telling me to not make this decision I’m telling you is the smart decision. Can you write that down?” We have a paper trail on that.

Brian: Jane Paris talks about this a lot, about keeping a decision log, not to be able to stab you with the ‘I told you so.’ It’s not so much that. It’s to measure how are we making decisions, and are we getting better or worse at it?

John: Yeah.

Brian: I don’t know if that’s what you’re talking about.

John: Yeah, and that’s the thing, we’re talking about everything we do is an experiment, but we need to learn from every experiment, and we need to be like, “Hey, man, we’ve done that. We tried that before. Remember what happened?” “Oh, yeah.” And then the other thing with dateless is ship date is not the end date of any given project because everybody has testing.

Some places, it’s different than production, but testing is never as complex as production. Your testing environment is never as complex as production. Your test users are never as diverse as your real world users. So, ship date is not the end point of any given product process; it’s the middle. And it’s like, oh, right, so like, time my timelines have on them, so here’s the ship date, and then here’s three to six months where we might have to pick up this project at any given time to fix stuff we found out only after it shipped.

Don’t just schedule in, oh we’re going to move on to this next thing. Like, that’s just… you won’t be able to [laugh]. And then we talk a lot about the chaos that software is past complex and into chaotic. So, that means that we can’t predict the outcome of any given action. We can’t. Any given code change, any given UX change, there is no conceivable way to predict the outcome. We can take bets, but we’re only going to be probabilistically right. And that mindset shift is hard for people, especially if they don’t come from software engineering. We have a lot of cultural issues around that, but gradually people have accepted that.

Brian: Yeah, are you familiar with Thinking in Bets, the Annie Duke—

John: Yeah.

Brian: Yeah.

John: Moving that culture to, kind of, what I’m able to [use 00:42:15] is to talk about installing that culture. Installing that culture across the organization, has been. Sales hasn’t a hundred percent bought in, but pretty much everybody else has bought in. And if you’ve ever been a salesperson—this is the other thing I’d say for product people, go try and be a salesperson. Understand how psychologically hard that is.

To me, it’s psychologi—someone who likes control—I mean, I’m a product manager, I like control—the amount you don’t control as a salesperson is so incredibly psychologically debilitating, I’d say. You have to develop these, like, worldviews to basically lie to yourself that, oh yeah, this next call I make, it’s totally going to work. This next email, it’s totally going to work. The confidence you have to maintain that’s kind of divorced from reality to do sales well shapes a lot of the conversations with them, and I think shapes a lot of the conflict between product and sales. It’s really, really, really hard to do sales, psychologically.

And you always want to believe that the next call, the next thing, the next feature is going to change your life. And if you’re on a commission, like, a commission B2B sales person, literally that next call, that next feature could change your life. It could be a life-changing amount of money, right? I’ve written seven-figure commission checks to salespeople [laugh] in previous jobs. Like, it could literally do that. And so, that worldview is so different than a product worldview. Just go, do it. Just be a salesperson [laugh].

Brian: [laugh]. I love it. I have a couple kind of wrap up questions here for you. This has been so insightful. Thank you for sharing all of this. First question: we’ve talked so much here about kind of the bread-and-butter parts of, like, the tactics of doing product management and requirements, or sprint planning and all this stuff, and we’ve talked so much about the people part of this, the soft skills, the non-analytical parts of doing product, the relationships, value, understanding people’s cultural fra—like, how does culture impact their framing of how they see the world.

I don’t know if you came out of a data lineage. I mean, you talked about working at Arc’teryx and stuff like that, but this isn’t how a lot of data product managers see the world. And have talked to so many of them. They’re coming out of logical thinking, analytical framing, type 2. Like, why would you pos—why would this possibly not be valuable to you? Like, just look at the numbers. Like, the evidence is staring you in the face, and you’re talking about all these other—you have, these other skills, John, that I think are really valuable. Did you actively go seek to get these skills, humanities, or, I don’t even know what you studied. Did you—

John: Yeah so, my academic background is history, political science, and economics, so I went to a school, one of the few in North America that does like the political economy of finance, and that’s where it came together for me that value is subjective. They’re teaching, like, political economy of finance, like, you think that’s going to be very, very, very data driven, right? But it’s not because the value of money is subjective. And so, I have this historical background, this humanities training on what is value, and how is it created, and how does technology work for the world. And I’ve always been obsessed with the history of technology, like, from the time as a little, little, little kid, and understanding how the technologies very often existed decades or centuries before it was adopted. Why?

Decades of my life, I was obsessed with that question. Why didn’t the Roman Empire skip straight to the Renaissance? Why didn’t China come to dominate the world, even—because they developed all the technologies: the rudder, the compass, the printing press, all the technologies Europe used to dominate the world—why didn’t they dominate the world? And so, asking those questions about, like, why does technology have impact, how does it have impact, that’s where all this stuff came from. That’s my obsession.

It’s like, why does it work and not work? And then just getting thrown into it. I was the very first product manager at Arc’teryx. I was the first person with that title trying to figure out how to do the job with no outside insight, like, with no outside input. Almost nothing had been written about it when I started doing it. So, it’s like, how do you do this? How do you create a successful product?

And so, just a lot of trial and error and experimentation, that’s how I got to it. And I think those data guys would really, really benefit from reading, you know, behavioral economics, Thinking Fast and Thinking Slow is one of the books that’s on my, like, work culture list. Like, “Hey everybody, read this book.” [laugh]. Understand how this works.

There’s another book called Warfare in the Information Age, which got me thinking about, like, how does data change behavior? It’s a study of how, basically, smart weapons came to be. But like, what happens when your amount of data to solve a problem goes up exponentially? What can you do? What behaviors do you need to change to do things? That’s a huge impact on me.

Brian: Great. I’ll put those in the [show notes 00:47:18]. I appreciate you sharing that because that was going to be my question was, is this something that—someone’s actually like, yeah, I see John has a skill that maybe I could develop. How would I develop that? I have a stats background. I went through math, or data science, or whatever. I have highly technical—these are less technical kinds of things, so where would I go? And so, I appreciate those recommendations. Any others you’d throw in there, people you follow, podcasts you listen to, anything like that, that might help open that world up?

John: So, I think I counted. I’m subscribed to, like, 10,000-plus accounts on social media—

Brian: [laugh].

John: —and I tell everybody who hires me, like, I’m going to spend hours out of my day, every day, just sitting and reading, and it won’t look like it’s related to what we do, and that’s just the job, right? Because you just don’t know where inspiration or knowledge is going to come from. So, there’s an old description of a product manager, like, oh, they come across as the smartest person in the room. Well, how do you become that person is, you just expand your view, expand the amount of information you consume as widely as possible, everything. I mean, I’m listening to the economists in NPR, and all across the political spectrum because my job is politically, all the sort of news media to understand the context.

I read a ton of history. I read a ton of technological history. I read, like, the top science books, five or ten top science books in the bestseller list every year. And then I just go and deep dive down on stuff that I don’t know about. I really dislike not knowing about things [laugh].

Like, I’m like, oh, there’s a whole branch of knowledge. Like, recently, I’ve been learning a lot about poker and magic, and it’s like, wow, my job is a whole lot like poker, right? It’s all about bets, and small bets, and, like, we talked about, the bet book. Yeah, okay, but I have a lot more unknowns. And then magic is, like, it reinforces the message of a book, like, Thinking Fast and Slow, which is about, like, people have consistent ways that they misinterpret the facts that are in front of them.

And that’s so important to UX design and thinking about, like, what the heck went wrong? That’s probably the question I ask more than any other one at work because, like, why are some customers super happy and some customers not? What is the difference between those two groups of people? Is it culture? Is it time? Is it mental ability? Is it the size of the screen they’re looking at my product on? Is it[laugh]—what variables can I define and rule out, right? And then what data sources do I have to answer all those questions? But it’s just the normal product manager thing, like, constant curiosity about, like, what the heck is happening and why is it happening.

Brian: You have a wide space of information you’re consuming. I think that’s where fresh ideas come from as well, when you have a diverse pool, and you’re not surrounding yourself with the same information. Or everyone thinks the same way, that diversity is really important to generating new things because you have more places to draw inspiration from.

John: And even more, like, after you’ve had the inspiration, it helps you justify it.

Brian: Yeah.

John: Like, why do you have confidence that this is a good idea? And you’re like, “Oh, somebody applied this same idea, so I made an analogy between my product and just-in-time inventory systems the other day.” Hey, what we’re doing is this, just-in-time inventory, just-in-five inventory transformed the following 20 industries over the last 50 years. It can then transform your industry. I gave a whole presentation on that. And they’re like, well, why does a software product manager need to be reading about how, like, warehouses work at Toyota? You just don’t know, right?

Brian: Yeah.

John: Until you do.

Brian: Yeah.

John: So.

Brian: John, this has been great. I want to ask where people can get in touch with you, if that’s okay. But before I do that, just kind of a closing question, is there any advice you’d give yourself ten years ago, if you had the time machine and could go back, about how you’re doing product or anything in this kind of product management space for analytics?

John: Don’t trust your own instincts. Really, really, really question everything you believe. Ten years ago, I burned probably 5 million bucks for the product resources at a company on a product [line 00:51:29] that flopped because I thought it was cool, and I didn’t spend the time going out and talking to hundreds of customers. I relied on my sales guys, who were in that region to tell me the truth, and I didn’t have a good enough relationship with them to understand when they were just, like, telling me what they thought I wanted to hear. And so, that really, really, like—hey, trust your own instincts to come up with an idea, but then test the crap out of it.

Really, really, really, assume that you’re wrong and you just can’t see it. That’s the biggest thing I learned. And, like, it’s a good—discarding what you think is a good idea is the single most valuable thing you can do as a product manager. I had a really, really good idea. I thought it was a really, really good idea, or the company had a really, really good idea, and we researched it, and we threw it away.

That’s the single most valuable thing you can do, right? Nothing is more expensive than building the wrong thing. I learned from that. That haunts me, it keeps me up at night, it’s cost me jobs in the past. And I’m like, no. I just—like, every time I do it, I test it.

Brian: It’s the second time I’ve heard that. I mean, I think I talked about this on my last episode. I can’t remember. I was listening to a podcast with the head of Google X, I think, and I think they work on these tough tech problems, right? The problem has to be really messy, really hard, but it has to have a monumental change if it’s going to work.

And they talked about this idea of one of the largest moments in the company is when the head of a project or a product or a service that’s being worked on stands up and says, “It’s time to shut this down.” There’s a standing ovation. A bonus is given to this person first stop and saying, “I’m not going to burn more of our resources.” What did we learn? We’re going to store our learnings on the shelf because it might be that we’re too early, but the point is, we learn something, but celebrating that willingness to say, I’m not going to invest more resources in this, but the culture really truly embraces that.

And to me, that’s how you foster innovation. If it’s all risk and, like, no one wants to raise their hand and say, this didn’t work, the organization has to support that, or else no one’s going to stand up—like, you are—and stay, “I wasted $5 million on this. This is not how you do it. I don’t want to do that again.” It takes bravery to stand up and do that, if the culture doesn’t exist for that. And I know a lot of places probably don’t have that, but I just wanted to call that out is, you’re not the only one [laugh], you know?

John: Yeah, I think that culture is changing broadly. In science, there’s now new journal called the Journals of Null Results, where they’re saying, “Hey, let’s stop only publishing papers that gets really amazing, statistically significant results,” let’s publish the papers to get null results. And that’s been a theme throughout my career, is that, like, null or zero is actually one of the most important numbers in any part of your business. Constantly be focusing, not on the big—oh, we grew a hundred percent. This had this.

This had zero, or this had negative, that’s a way more important number, a lot of the times, than the huge positive number. And the same thing with every experiment. Like, that experiment failed. Awesome. You did your job. You failed at the pre-MVP stage. Awesome.

I had a company I worked for, it was a Fortune 500 company, and I was a product manager, and one of the guys on the board had been the head of product development for Target, like, their private lines at Target. And I said, “Hey, like, the assumption I’m working for is a 20% success rate is good for product.” And he’s like, “Oh, yeah. At Target, we assumed at least 80% of the SKUs we launched and took to stores would fail to make a profit.” Right? Was Target private lines a bad business and not a profitable business? No [laugh]. They accepted that failure rate, right?

And I think the higher your acceptance for, like—or you can have a really successful business with a super high failure rate for products. Different for every business. But fail fast—the famous thing—fail quickly and, like, cel—yeah, absolutely, celebrate failure, and you know, have those retrospectives to be, like, what failed, and why, and what have we learned from it? Because learning it’s, you know… different context, but a military guy I was working with, he was like, “Yeah, learning only occurs through repeated, humiliating failure.”

Brian: [laugh].

John: And if your goal is to be a product manager, your goal is to be a learner. If your goal is to have a product-led organization, that means a learning organization, and that means accepting repeated humiliating failure.

Brian: Yeah. I bet the humiliating part is pretty important in that statement. I’m just thinking about—just, like, backtracking my 45 years or whatever of existence, and I’m like, yeah, the humiliating part is definitely correlates highly with the do I remember the learning part [laugh].

John: With deep learning, right? The humiliating is what creates the deep learning.

Brian: Totally. Totally.

John: Exactly. And it’s like, “Okay, that feels horrible,” and then you’re like, “Okay, I accept that that feels horrible,” and then reflect back on yourself. But that means I learned something super important. I’m still alive. We’re still in business, so we learned something super important.

Brian: John, this has been great. Where can people follow you? LinkedIn? Are you on social media?

John: Yeah, LinkedIn is the best place.

Brian: Cool. And if they want to get in touch, is that the best place to do that?

John: Yep. Just reach out and I’ll accept.

Brian: Cool. Sounds good. Well, thank you so much for coming on the show to talk about LabStats and your career. We talk about Arc’teryx. I like those jackets, and I don’t even—I’m not, like, a mountain person, but, like, I kind of want one of these jackets. Like [laugh], they’re pretty dope.

John: Go try one on.

Brian: Okay.

John: Yeah.

Brian: Try it on. Okay, I will do that.

John: That’s how we designed everything there was, like, if it doesn’t pass the ‘if you try it on, you’ll buy it’ test, we won’t ship it.

Brian: Oh. Well, maybe I shouldn’t try it on because I’m going to own a bunch of jackets [laugh].

John: [laugh]. Yep.

Brian: Awesome. Well, thank you, John, so much. It’s been great to talk to today.