Eric Weber, Head of Data Product at Yelp, has spent his career developing a product-minded approach to producing data-driven solutions that actually deliver value. For Eric, developing a data product mindset is still quite new and today, we’re digging into all things “data product management” and why thinking of data with a product mindset matters.
In our conversation, Eric defines what data products are and explains the value that data product managers can bring to their companies. Eric’s own ethos on centering on empathy, while equally balanced with technical credibility, is central to his perspectives on data product management. We also discussed how Eric is bringing all of this to hand at Yelp and the various ways they’re tackling their customers' data product needs.
In this episode, we also cover:
- What is a data product and why do we need data product management? (01:34)
- Why successful data product managers carry two important traits - empathy and technical credibility. (10:47)
- A discussion about the levels of problem-solving maturity, the challenge behind delivering solutions, and where product managers can be the most effective during the process. (16:54)
- A look at Yelp’s customer research strategy and what they are focusing on to optimize the user experience. (21:28)
- How Yelp’s product strategy is influenced by classes of problems – and Yelp’s layers of experimentation. (27:38)
- Eric reflects on unlearning and talks about his newsletter, From Data to Product. (34:36)
Quotes from Today’s Episode
- “Data products bring companies a way to think about the long-term viability and sustainability of their data investments. [...] And part of that is creating things that are sustainable, that have a strategy, that have a customer in mind. And a lot of these things people do - maybe they don't call it out explicitly, but this is a packaging that I think focuses us in the right places rather than hoping for the best.”- Eric Weber (@edweber1) (02:43)
- “My hypothesis right now is that by introducing [product management] as a role, you create a vision for our product that is not just tied to a person, it's not just tied to a moment in time of the company. It's something where you can actually have another product manager come in and understand where things are headed. I think that is really the key to seeing the 10 to 20-year sustainability, other than crossing your fingers and hoping that one person stays for a long time, which is kind of a tough bet in this environment.”- Eric Weber (@edweber1) (07:27)
- “My background is in design and one of the things that I have to work on a lot with my clients and with data scientists in particular, is getting out of the head of wanting to work on “the thing” and learning how to fall in love with the customer's problem and their need. And this whole idea of empathy, not being a squishy thing, but do you want your work to matter? Or, do you just write code or work on models all day long and you don't care if it ships and makes a difference? I think good product-minded people care a lot about that outcome. So, this output versus outcome thing is a mindset change that has to happen.”- Brian T. O’Neill (@rhythmspice) (10:56)
- “The question about whether you focus on internal development or external buying often goes back to, what is your business trying to do? And how much is this going to cost us over time? And it's fascinating because I want [anyone listening] to come across [the data product] field as an area in motion. It's probably going to look pretty different a year from now, which I find pretty awesome and fascinating myself.”- Eric Weber (@edweber1) (27:02)
- “If you don't have a deep understanding of what your customer is trying to do and are able to abstract it to some general class of problem, you're probably going to end up building a solution that's too narrow and not sustainable because it will solve something in the short term. But, what if you have to re-architect the whole thing? That's where it becomes really expensive and where having a product strategy pays off.”- Eric Weber (@edweber1) (31:28)
- “I've had to unlearn that idea that I need to create a definitive framework of what someone does. I just need to be able to put on different lenses. [For example] if I'm talking to design today, these are probably the things that they're going to be focused on and concerned about. If I'm talking to our executive team, this is probably how they're going to break this problem down and look at it. So, I think it's not necessarily dropping certain frameworks, it's being able to understand that some of them are useful in certain scenarios and they're not in others. And that ability is something that I think has created this chance for me to look at the data product from different spaces and think about why it might be valuable.”- Eric Weber (@edweber1) (35:54)
Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill, and today I have Eric Weber on the line from Yelp. Eric, how are you?
Eric: I’m doing great. How are you, Brian?
Brian: I’m good. Yeah, and you’ve got the word ‘data product’ up in your title, up in all your LinkedIn posts, and you’ve got a new mailing list, all kinds of data product goodness, and so that’s what I wanted to jam on today with you and get your feedback about this kind of—it kind of feels new kind of feels old, some of this is just terminology, but I think there’s a different approach when we think about product as a mindset and not just a thing, and how we build data-driven solutions and design them around customers. So, that’s why I wanted to dive in with you today about it, if that sounds good.
Eric: Sounds great. I’m excited to talk about it, and also, the best part about it is we’re still at the start, so there’s a lot of things to be defined, a lot of [laugh] things to discuss. And at this point, it’s like, we’re making hypotheses about what’s important; excited to dive in.
Brian: Let’s start there. What the heck is it? What’s a data product?
Eric: So… I think rather than being a brand new type of idea, data product, as you mentioned before, is a new story consisting of some older ideas. For a long time, we’ve—we have a lot of frameworks about how to create, develop, manage products that inform companies all over the place, not just in tech, but in all of our classic Fortune 500s, they have product management. Most companies also have data or some story around their data; a lot of times that story is not very pretty. [laugh].
And over the last eight years, especially, companies have built a lot of things with that data. They’ve built dashboards, models, data assets that help them do things, and data product is really trying to merge and almost smash those two things together and see what comes out. And I think that’s the most interesting part is that it’s a change in motion. It’s not something that we’ve fully settled on.
The most important thing for me is that thinking about data product brings companies a way to think about the long-term viability and sustainability of their data investments, which is data as a product is really supposed to be a way to think about building products from data, and part of that is creating things that are sustainable, that have a strategy, that have a customer in mind. And a lot of these things, people do, maybe they don’t call it out explicitly, but this is a packaging that I think focuses us in the right places, rather than hopes for the best, which is kind of a [laugh] general sentiment that I’ve seen in many companies.
Brian: Does this idea of a data product manager or data product management, is it important to have that as a defined term, a defined role? Is it more about just having the behaviors happening and the actions happening? Do you have any strong opinions about that? Especially when we think about this outside of the digital natives? And this is a divide I want to talk about a lot on this episode with you, but is that a meaningful distinction to have that role?
Eric: Rather than just look at my opinion, I think it’s important to think about—since I started even writing the newsletter around data product, I’ve had a lot of people come to me from many different types of companies, from government agencies to more traditional blue-chip type of companies that would not be digital natives, and talk about why it matters. And I think a big part of why it matters is it defines something as an important area of investment. And I think that area of investment up until now was identified as data—data science, data analytics, whatever it might be—but that produced a lot of one-off solutions, a lot of things that companies maybe derived some value from but that value was not—like, didn’t keep pace with the change in their strategy, the change in their growth potential.
So, I think this is where it comes in, which is to say if we’re going to invest in data, we also need to invest in product managers who think about managing our data investments over time. And so I think the value there is, if you don’t explicitly create this type of role, the behaviors and actions that you hope for I think are much less likely to take place. Or they take place informally or sort of on the edge of someone’s job. A data scien—you might get lucky and have a data scientist who says, “Hey, I’m happy to also be a product manager and to think about strategy,” but if you want to bet your data [laugh] approach at your company on that happening, I wouldn’t take that bet. And I think this is the reason that you’re starting to see this idea expand across different parts of industry.
Brian: If you were at a non-digital company and let’s say you’re advising a CDO, or CAO or somebody like this who has a lot of executive responsibility to bring value out of their data assets, why should I bring on data product management? What am I going to get for that? So, I already hired this data science team. They were really expensive, we’ve thrown a ton of money at that, and we’re in the cloud and all this—we’re not on-prem anymore, and we did all this stuff. Why do I need this? What do I get? [laugh]. Tell me what I get, Eric. [laugh].
Eric: First of all, it’s a great question. I think we want to walk a bit back to this idea what happens in the absence of doing this? I think it’s a powerful statement. Most non-digital-native companies have had data science around for a while now. At least when I say ‘a while,’ it’s all relative, right? Everything we talk about is relative.
But they’ve had a few years to see what happens, and the vast majority of them can point to a number of high impact, fairly damaging stories about what happened when they put all of the burden onto their data science team. And the reality is, is that companies ch—like people come to companies, they leave. This is true of executives, it’s true of individual contributors. A huge risk that you introduce if you don’t have a strategy for that product that the company can own, then when people leave, your strategy for that product is also very likely to walk out the door.
And what I’ve seen, and I think it’s important to step back and say, like, “We’re learning as we go on this.” My hypothesis right now is that by introducing this as a role, you create a vision for a product that is not just tied to a person; it’s not just tied to a moment in time of the company, but it’s something that you can actually have another product manager come in and understand where things are headed, potentially introduce their own variety or version of what that looks like, but I think that is really the key to seeing the 10 to 20-year sustainability, other than crossing your fingers and hoping that one person stays for a long time which it’s kind of a tough bet [laugh] in this environment.
Brian: A linchpin—yeah. Especially when there’s a job shorta—there’s a talent shortage and [laugh] all of that. Yeah, no, I understand. Does the level of quote, ‘data maturity,’ or software, or digital-ness of the company have any bearing on whether or not this is important? Or that’s irrelevant, the scale you’re at, the quality, the maturity level?
Like, if you were starting in, like, we’re an old-fashioned company, we’re a traditional company. We’re about to jump in and invest. We’ve heard we really need to be doing more with this. Do we grow into this need of a data product manager, or is that a place to start with?
Eric: That’s a great question. I think—I mean, just—I think most people who are listening probably can look back and have heard the story of, hey, we know that our peers are investing in data so what’s our first action going to be? Hire a data scientist to tell us what we should be focused on and to figure out how to derive value. Put another way, they kind of jump in without a strategy.
Eric: Right? And sometimes you get fortunate, where that—like I said, that data scientist may come in and they may have a really good idea of how to tie this to business value. But we’re seeing a divergence, and data scientists are becoming more specialized in particular areas, and they want to come in and work on the things they want to work on. And they can definitely demand that primarily because the market is so crazy. There’s just no—and so if you don’t have a strategy—and it doesn’t have to be something that says, “Here’s where we’re going for the next five years,” but someone should own that.
And that could potentially be someone who’s already at the company, which is completely fine. You don’t necessarily have to go bring someone else in. But the idea of having a plan and having a strategy for how you’re going to invest in particular data products is critical. A really important component here is a lot of companies talk about having a data strategy—I’ve particularly noticed this that non-digital-native companies, they’re like, “We need a data strategy.” I actually think they probably need a data product strategy, which is, “How are you going to derive specific value out of parts of your data assets?” Rather than have this holistic—which is what a CTO might focus on—this holistic idea of data transformation and value.
Brian: This does get thrown around a lot, and I’ve heard some really good framings that it’s like, “Well, it’s really, let’s figure out what the business strategy is, and then how data serves that.” You know? [laugh]. There’s lots of different framings for that, and I understand what you’re saying there. How do we identify the skills that we want, particularly for someone who’s highly trained in a technical area?
Think empathetically. So, my background is obviously in design, and one of the things that I have to work on a lot with my clients, and with data scientists, in particular, is getting out of the head of wanting to work on the thing, and learning how to fall in love with the customer’s problem and their need. And this whole idea of empathy not being a squishy thing, but it’s like, do you want your work to matter? If you did all this great NLP work, do you want it to matter, or do you just want to just write code or work on models all day long and you don’t care if it ships and makes a difference?
And I think good product-minded people care a lot about that outcome. So, this output versus outcome thing is a mindset change that has to happen. Do you agree with that? And is this something we can develop out of anybody? Is it something we train into people? Do you agree that’s what’s needed to do that well?
Eric: Again, I think your questions are all spurring longer posts, discussions—
Eric: —in my mind, which are good, which are very good. I think when it comes to this space, we have—I mean, just look at data scientists: the reason that they are data scientists are all over the place. Some of them are just fascinated by the architecture and the model building, really could not care less about what business they’re doing it in. Others are very motivated by the business that they’re operating in—
Eric: —and they try to figure out what is the best approach for the business that I’m operating in. Again, I think it’s rather than the role specifically; it’s do you have people who can represent the interests and what’s good for the business, and also have credibility with the technical crowd? I think the reason I’ve seen a lot of technical people become successful data product managers—this is certainly true at Yelp—is that they have this level of credibility built with engineers, data scientists, whoever may be, and it creates trust. And it creates trust that you may be talking about business outcomes but that you understand and can empathize with where they are.
And so I think it’s this dual empathy that’s really critical. Can you understand what it’s like to be in a data scientist role and experience those constraints, and can you also understand what it means to make a decision on, do we invest in this area or this area to support the investor-promised growth of the business and I think it’s that dual empathy? Not everyone is going to have it, either. You kind of have to choose. You may have people who over-index in one space who are motivated to index higher in another. You’re probably not going to find all the unicorns that operate in both because I don’t think they really exist. [laugh].
Brian: The design world has the same issue, you know? A lot of talented individual designers I know, they love the craft of doing that. “I went to school for this. I shouldn’t have to explain my job and why I do it, and you should just trust that it’s right.” And it’s about the craft.
But if you want to be really successful with that, you need to understand it from the engineering’s perspective, and you need to understand that, no, we can’t predict this because the data is a complete mess. The accuracy matters. Wrong decision-making has business consequences. So yeah, that chart’s really nice that just tells the answer, the magic answer to the customer, but there has to be some meat and potatoes behind that, some solid foundation behind that.
So, you know, I agree with you, what I’ve seen that works best is when all these roles and can be empathetic to the other roles there, and they understand enough. They don’t need to know how to code they don’t have to build a model, but a designer needs to understand those pains and challenges in order to design really good solutions. And vice versa, I think the data teams need to understand the customer, the user’s perspective, and how there’s irrational things that happen, and emotions come into play, and, “My confidence in my need is a huge at this company, I’m being replaced by machines.” Like, this stuff is all really real. It’s not fake.
It’s real stuff, and just the attitude of, “It’s not my problem if they don’t understand it,” which still exists. I mean, this just came up again this week on LinkedIn and someone mentioned a comment on my post, and it’s just like, “It’s not my problem if they’re bad at their job.” That was the reply. And I don’t know, it sounds like there’s more commonalities to some of my past, [laugh], you know, when I was a staff designer somewhere. I don’t know, do you agree with that, like, we need to understand each other better? [laugh].
Eric: I think there’s a lot of people who, in a large meeting will talk about the need to break down silos and empathize—
Eric: —and then the minute they get out of the meeting, will start being like, “It’s not my job.” And this is really important because if you think about what are the incentives for people to do that, and care? It’s great when people are intrinsically motivated to do this, and break down silos, and work across boundaries, and understand each other. You’re going to get lucky in some cases, but what about in the cases where you’re not? How do you—
Eric: And I think that’s where people have operated on hope. Hope is not a good strategy here, and I think this is where product managers, you can make them accountable for doing this type of work. And I think it actually introduces a degree of accountability to be able to say, “This person is responsible for doing this.” And that, I think, is a value-add. And it sometimes looks like in different forms, depending on, like, experimentation, analytics, whatever it may be. I do think there’s a lot of value in having that person be accountable for those types of actions.
Brian: I wanted to jump over to some questions about problem-solving and problem discovery and get your take on that. So, I see three levels of your problem sleuthing-ness. There’s, like, giving people what they asked for is, kind of like, level one. Sometimes that’s a win because they know how to form the statement really well and they know how to ask a question really well.
Step two is, I work with someone to understand what they need, and specifically, I’m good at getting to the unarticulated problem through research, inquiry, shadowing somebody, understanding their job.
Level three is I anticipate needs before they even occur because I know the data well, I know the business well, I’ve talked to them for years, I can see, “Oh, I can help the accounting department with this. I’m going to set up a meeting and talk to them about this.” Those, kind of, three levels of maturity are there.
Do you think… do you agree with that? First of all, do you agree that that structure exists? And is the big problem with low adoption of data products in our industry because we’re mostly at level one here, which is giving people what they asked for, and we’re not doing level two, which is getting out and understanding the unarticulated needs and knowing, “I want machine learning; can you give me an algorithm that will do this?” It’s like, it’s loaded with solution. You know, that statement needs to be unpacked. Is that why we’re not doing well because the industry does not do well with delivering great solutions.
Eric: The hierarchy that you mentioned, the three levels, I think resonates because it does tell that story of different ways that we can add value. My guess is—and this is, you know, again, it’s based on experience rather than, like, “I have a broad survey of information.” And I think this—like, with level one, I think that’s people’s generally—that’s their first approach, which is to say, “What do you need? Literally, write down what you need, and then we’re going to go build it.” And when you have data science going directly to an engineering team—which is a dynamic that certainly exists in many places—or data science acts as the engineering team themselves, they’re going to build the things that they hear about in the moment. And, you know, strategy sounds great until someone’s pinging you on Slack late at night being like, “This is broken, fix it.” Right? Like, this is a—
Brian: Is it done yet? Is it done yet?
Eric: [laugh]. Yeah. Right? So, strategy can blow up, and I think there’s a few issues. One is that, like you mentioned, we’re anchored very much at level one, and it’s hard to sometimes get out of that because that extra requires having a strategy and what I would say, like, a reason to say no to certain things.
Eric: Because if you don’t have a direction that you’re going, it’s very hard to decline things, even if you don’t think they’re valuable because what else are you going to do? The other issue, I think, is that we go too quickly to level three, without having done a lot of the legwork to actually understand our customer. Like, people talk to—like, a good example: a few years ago, I was at a company and we were doing—there’s a lot of discussion around, “We need to understand the customer.” So, this person walks in; two days later, comes back, she’s like, “All right. I’m ready to go.” Like, “How do you—what did you do to understand the customer?” Like, “Oh, I walked over to the desk of these two people.” I’m like, “Okay. [laugh]. Does that… do you think that tells you what you need to know, in order to anticipate their needs in the future as well?”
And I often think we go off of anecdotes, we go off of too many small experiences and generalize those to make principals about what’s going to be important two or three years from now. And so I think we don’t spend enough time. And I think the key to moving from level one to two to three is to not only understand your customer at a moment in time but to understand how their needs are changing over a period of time. It’s not enough to drop in every six months and say, “Hey, I’m back.” And this is where product manager who’s very effective can keep the pulse on what those changes look like and how the customer is evolving.
Because that’s the only way I think you really get to level three is to understand how people’s needs—
Eric: —change. And if you just look at a slice in time, which is what a lot of customer needs frameworks look like, [laugh], it’s very hard to anticipate how they’re going to change over time.
Brian: Yeah, yeah. At Yelp, do you involve—this is what you know, the user experience discipline has talked about forever, and there’s still companies that are really fighting to do more customer research as foundational, and not just jumping design, which is tooling, and solutions, and making stuff. You know, the research is about really that problem discovery, understanding the need. And we all hear—we talk about it a lot that we won’t have to spend so much time making stuff and making wrong stuff if we understand the need better. But it doesn’t always happen. And I’m curious, do you partner up with UX at Yelp to do this kind of research, or do the data professionals do that on their own? Do they go talk to the marketing team by themselves and do that kind of research? Like, how do you do it?
Eric: I think in a very large, mature data product org, it’s probably a combination of all the things you mentioned: it’s UX, its data product. We’ve started from the place of trying to get data product managers as close to the customer as they can be. And so rather than rely on any layer of translation, it’s, “Hey, we want you to spend x number of hours a week engaging with different parts of the customer base, whether it be marketing, or sales, or product managers who are trying to run experiments.” I don’t think there’s an easy replacement for hearing direct feedback. What I try to focus on is then learning by an accumulation of evidence because people tend to be fairly reactionary and they’ll have a very insightful conversation, and then immediately think, “Okay, this is altering the whole roadmap.” From this one conversation.
So, I think it’s more about how do we accumulate evidence over time across these different areas and understand customers over time. And UX is definitely a part of that. I think we probably could do better at Yelp in that space, which is to really think about what the technical design looks like of our tools and user experience. I think for right now, it’s understanding just what they’re trying to do, the making sure that we have the right things in place. But I think there’s a place for, when you talk about optimization of user experience, we’re partnering with our design team to focus on some aspects of our experimentation platform, now. And once you get there, and you understand the needs really well, you can start to optimize certain parts of the experience so that people start to do [laugh] more useful things and actually find it to be a pleasant experience instead of something that they have to hammer through and painfully walk through. [laugh].
Brian: Yeah, yeah, yeah. How does that interaction work at Yelp? Is it challenging? Is it mostly just both sides find it really beneficial? What’s it like?
Eric: I don’t know if it’s developed deeply enough to say like—to describe the whole thing. I can tell you that when it comes to experimentation, it’s really—like it started from me going to talk to our head of design and saying, “Here are the things we’re trying to do. This is why it matters for Yelp. These are the things we’re trying to do better that we don’t have the internal team expertise on. Would we be able to partner with people in the design org to help this?”
And so, I’d say right now it’s more dependent on a certain product and understanding when UX needs to come in. I do think if there were a—I would say in the near future, we’ll have probably much more of a technical product design group where they focus on working with our deeply technical products—that could be for developers working with data, it could be for data scientists—to focus on creating the best experience possible. I think this is probably an area that a lot of companies just are missing out on because, again, the silos.
Eric: I know I’m not a designer, and so I know a lot of things that I don’t know, and I go talk to these groups say, like, “Do you think there are things we could do better in this case for the customer?”
Brian: Right. No, you’re right. I mean, a lot of—when software companies, especially data software companies come to me, whether it’s data management, or storage, or insights, tools, they filled in that at the sale, it’s changing. They’re feeling it in the sales cycle, right, where the quote, “Enterprise buyer,” the head of VP of data science is the one that’s going to have to use this tool that’s supposed to save his team job. The tolerance level for crap is so much lower than it used to be.
Like my phone, I can do this on my phone. Why is this so complicated? Look at this thing I could do. I can look at a billion-row spreadsheet on my phone, this little startups app, and you’re selling me this, like, enterprise-grade tool that looks like it’s 25 years old. And I don’t know if you agree, but I think the tolerances for that stuff have changed. It needs to save me time, it needs to make my work better, and data scientists are users, too. Analysts, they’re users, too. You know?
Eric: Absolutely. There’s so many places where people are not putting up with the stuff that they used to. And I think it’s good. It also has spurred a lot of, I think, product managers, in this space especially, have to navigate a lot of build versus buy discussions, which back to your point that you just mentioned, we see a lot of solutions out in the marketplace, a lot of startups, data startups are generating things that we’re like, “Oh, that looks really—that looks ten times better than the thing we have. [laugh] why would we not just buy it?”
And like the question about whether you focus on internal development or external buying often actually goes back to, like, what is your business trying to do, and how much is this going to cost us over time? And that’s again, like—it’s just fascinating because again, I think anyone listening, I want them to come across this: this is a field and an area in motion. It’s not something we’re just—we’re describing it right now in, you know, August 2021. It’s probably going to look pretty different a year from now, which I find pretty awesome and fascinating, myself.
Brian: Yeah. I wanted to jump back to one of the th—we were talking about this kind of problem space definition and how we uncover these types of problems, and I wanted to ask you about, I know you have a large experimentation platform at Yelp. So, there’s this idea of, we’re designing for classes of problems, versus we’re designing a custom solution for this. A bill pay system that lets you write checks online to somebody, it’s very discreet and specific. It’s different than designing for a class of problems.
I think that’s really hard to do and I’m curious if you have any strategies for that because you need this infrastructure to do so much of the work, especially with advanced analytics, and predictive analytics, and machine learning, you need infrastructure which, it’s its own product in a way. Do you even think about it that way, like, classes of problems instead of discrete problems? Or do we start with a discrete problem, then let’s genericize it. Let’s make this a component that can be reused, kind of the software engineering mentality? How do you think about it?
Eric: I think if you listen to most people talk about their products, it sounds like they knew what the generic class and problem was, and they designed for it. In reality, they probably started somewhere. They started—so for us at Yelp, we started with an experiment in consumer. It was like, “Hey, okay, we ran a couple. That seemed to go well. We should run some more.”
And so the initial version of the platform—I don’t know if I call it a platform at that time—was really focused on supporting consumer experimentation. And we have a lot of things that are baked into that, which is, like, sample size is easy to hit; you don’t have an issue with power because we have hundreds of millions of monthly users, we can easily divert traffic and do what we want. So, I think the class of problem for us is actually defining what area of the business is this experiment affecting? Are we experimenting at the business level or the user level? Are we experimenting in our ads business, our local business segment, our consumer area?
So, I think the issue is that the class of problem is often tied to your strategy and the verticals you have in your business. This is a—I think it’s often where we have classes of problems, when you have an external platform or tool come in and say, “Hey, you should buy our solution,” the integration costs are usually on, how do we make their solution work for the classes of problems that we have at our company? There’s a lot of other stuff that comes into it, but I think that’s a major challenge.
Brian: Do you agree that the definition of—doing a good job of articulating the definition of those is a challenge? I mean, this to me gets back to this research thing. If you don’t have enough customer exposure hours, I don’t know how you possibly do that accurately without just guessing that, “Hmm, maybe marketing could maybe want to classify some things.” Like, these abstract out of the totally uninformed guesses. It’s the hope—I don’t know, but I think you need a lot of customer exposure time, which doesn’t seem to be normal because we focus on making: building, making, doing. [laugh]. What’s your take?
Eric: I think the classes of problems that you’re talking about are really critical for actually creating a useful product strategy for a data product. It’s like if you want to come up with a path forward for a product that’s going to really be persistent over a year, two or three years, the things that will remain stable are probably these classes of problems that you’re trying to work with. The specific form of them may look different over time, but you generally know that these are going to be thematic things that you’re coming back to, that your customers are trying to do, over and over again. And that, to your point, if you don’t have a deep understanding of what your customer is trying to do, and are able to—again what you said—abstract it to some general class of problem, you’re probably going to end up building a solution that’s too narrow and not sustainable because it will solve something in the short term, but what if you have to rearchitect the whole thing? And that’s where [laugh] it becomes really expensive and where having a product strategy pays off.
Brian: Sure. This is totally anecdotal, but I feel like I’ve I have seen a fair number of projects where it’s mostly about infrastructure, and all the theoretical things that one could do with it if one knew how, and it’s not my problem if they can’t figure out how to do that because it does support all those things. So, you end up with lots of plumbing, no faucets in the right place, pressure’s not quite right, maybe the water’s a little dirty. Is there a balance here? How do we find that balance between that and—because it’s easy to show progress and say, “Look at what we made. We stood up this thing in the cloud. We made this thing and now you can pull data from all these places.” And this business person’s, like, “And I get what for that? What can I do with that?” [laugh].
Eric: I think experimentation is a good example here. We have multiple engineering teams that work on it. One of those is focused specifically on the infrastructure: how do we log? How do we capture data? How do we actually move it through our pipelines?
There’s another engineering team that is entirely focused on what I would call more of, like, the customer experience: what do they actually go through when they set up an experiment? How do we get them to identify a minimum detectable effect or something like that?
And we have data scientists who are working at this layer, too, to say, “Hey, this is what people are trying to do.” So, I think you’re—if you only have the infra perspective, like, “Hey, if you knew how to do all this stuff and you knew how to operate from the command line, you could use this tool as well,” it’s probably not going to be a good thing. And so rather than have an entire team that’s supposed to manage everything all the way from the infra to the end-user experience, I think is pretty challenging because there’s this idea that, “Oh, I’ll just get a full-stack engineer, that way they’ll be able to do infra and design.” And I’m like, “No, that’s not”—
Eric: —“That’s not a thing.” Like, “No one can do that.” So, I think you really have to figure out, what do you need this engineer or data scientist to be really good at? And to build different layers into that support because if you just have a whole bunch of infra-engineers, you’re going to end up with some great infra and no one’s going to use your product. Similarly, too many frontend and everything is going to look beautiful, and when they click something, it’s not going to do what they want it to do.
Brian: Right. Yeah, yeah. No, I understand. Any—Eric, I just wondered, I want to get into your mailing list and how people can follow up with you. I’m curious if you’ve had to ‘unlearn what you have learned,’ and [laugh] have you had to change anything about your whole approach with this?
And I’m particularly asking because I’m stereotyping here, but I do tend to see highly narrowly specialized trained people, especially those with PhDs, they tend to look at the world in a very detailed, specific way because that’s what they know. And sometimes that means unlearning some of that and realizing the world isn’t that way. A lot of the rest of the world isn’t that way if you don’t work in that domain. I’m curious if that’s true for you, or just in your career, if you’ve had to change any of your perspectives about having a good career with this stuff and making [crosstalk 00:35:03]—
Eric: I’ve had to, I’ve had to blow up a lot of frameworks by which I see the world. And you’re totally right. I mean, most people in data are highly trained to do something, and part of what that does is it allows you to do that thing, but you also tend to bring that lens and that framework to try to interpret everything else. And you get frustrated when something doesn’t [blank] that framework. And you’re like, “They just don’t understand what I do.”
So, for me, it’s been really important to ask a question to people when I meet them and talk to them. I’m like, “What do you do?” And it’s really fascinating the types of responses you get because when you ask a fairly open-ended question like that, they also tend to introduce how they see the world, how they see the business, what they think they’re pushing for. And I’ve had to really unlearn that idea that I need to create a definitive framework of what someone does. I just need to be able to put on different lenses for like, hey, if I’m talking to design today, this is probably the things that they’re going to be focused on and concerned about.
If I’m talking to our executive team, this is probably how they’re going to break this problem down and look at it. So, I think it’s not necessarily dropping certain frameworks; it’s being able to understand that some of them are useful in certain scenarios, and they’re not and others. And that ability is something that I think has created this chance for me to look at the data product from different spaces and think about why it might be valuable.
Brian: Yeah. I’m fully with you there. I think that’s great, great advice. And on the subject of advice, you’ve been publishing a ton on LinkedIn. And that’s why I actually reached out to you. So, you’re super active on there. We’ll put your profile link in there. You have a new mailing list, though, right?
Eric: Yeah. I started writing newsletter called From Data to Product, I think maybe four months ago now. And I think for a long time, I had tried to figure out, you know, I need to do something on top of LinkedIn. And it wasn’t until I finally found there w—there are so many newsletters about data science; like, so—like, many, many, many. And once I really figured out that I think there’s something here that needs to be shared around data product, that’s when I felt like I had a specific area that I could discuss and contribute in.
And so I’ve started writing. I wouldn’t say that there’s a strategy to the topics, they’re very often reflective of the things that I’m learning and thinking about. And they’re generally trying to spur conversation rather than be a, like, 20-page deep dive into, this is a framework for how to think about this. And I think the best part has been that it creates interesting conversations. It also identifies holes and gaps in my own thinking and approach to things. So, it’s been awesome. I don’t know—you know, managing the newsletter and writing is its own time investment, but I—it’s creating interesting conversations, and that’s kind of my proxy metric for, “Is this useful to keep doing?” [laugh].
Brian: Yeah, yeah, yeah. Where can people find that? What’s the URL?
Eric: It is on Substack, and it’s called just ‘ericdataproduct.’ So—
Brian: ‘Ericdataproduct?’ Okay.
Eric: Yep. And again, we can include the link afterward. But—
Brian: Yeah, yeah.
Eric: —yeah, there’s a lot of I think writing, I think doing what you’re doing with podcasts, like podcasting. We’re all trying to create conversation and help people think about things in a different way. And so this is, even the questions you’ve asked today, they’re going to stick in my mind for [laugh] the rest of the day, any be like, “Wh—hmm. That was, it was deeper than I even thought we were talking.”
Brian: [laugh]. Well, good. I’m hoping it’s stimulating things. And I’m with you; I’ve been writing for five years and I write to figure out, to clarify my thoughts, and why do I actually think it—like you asked, “What do you do?” It’s like, “I had to write for five years to figure that out.” I think I still am.
But it helps you clarify your thinking and it’s not premeditated. I’ve learned so much about the power of writing, and how it’s not you think it all up and then you dump it on the paper. It’s like, no, actually, you start writing, and then you figure out what you’re going to say as you’re writing. It’s comple—it’s almost the way design happens. You do design, and then you figure out what’s needed in doing it. You don’t premeditate it upfront, and it’s a different—it’s a making-first kind of way of thinking, and I don’t know, I find it really helpful for my own [crosstalk 00:39:38].
Eric: I do too. Like, sometimes I don’t know what I’m going to write about when I start, and I just have an experience that I’ve gone through that is important and I try to think about what really mattered from that experience? And then I often have something I can share. And that’s… and people like, “Well, do you plan, like, weeks in advance?” I’m like, “No.” [laugh]. Like, that would be nice if I really thought I could do that, but no, it’s very reflective of experience.
Brian: [laugh]. Yeah. No, that’s great. Well, Eric Weber, thank you for coming on Experiencing Data and sharing these thoughts. Eric Weber from Yelp. I’ll put all his links up on the [show notes 00:40:12], and it was great to have you. Thanks for coming on and talking.
Eric: Thanks so much for having me, Brian. I appreciate it.
Brian: Take care.