141 – How They’re Adopting a Producty Approach to Data Products at RBC with Duncan Milne

Experiencing Data with Brian O'Neill (Designing for Analytics)
Experiencing Data with Brian T. O'Neill
141 - How They’re Adopting a Producty Approach to Data Products at RBC with Duncan Milne
Loading
/

In this week's episode of Experiencing Data, I'm joined by Duncan Milne, a Director, Data Investment & Product Management at the Royal Bank of Canada (RBC). Today, Duncan (who is also a member of the DPLC) gives a preview of his upcoming webinar on April 24, 2024 entitled, “Is that Data Product Worth Building? Estimating Economic Value…Before You Build It!”  Duncan shares his experience of implementing a product mindset within RBC's Chief Data Office, and he explains some of the challenges, successes, and insights gained along the way. He emphasizes the critical role of understanding user needs and evaluating the economic impact of data products—before they are built. Duncan was gracious to let us peek inside and see a transformation that is currently in progress and I’m excited to check out his webinar this month!

Highlights/ Skip to:

  • I introduce Duncan Milne from RBC (00:00)
  • Duncan outlines the Chief Data Office's function at RBC  (01:01)
  • We discuss data products and how they are used to improve business process (04:05)
  • The genesis behind RBC's move towards a product-centric approach in handling data, highlighting initial challenges and strategies for fostering a product mindset (07:26)
  • Duncan discusses developing a framework to guide the lifecycle of data products at RBC (09:29)
  • Duncan addresses initial resistance and adaptation strategies for engaging teams in a new product-centric methodology (12:04)
  • The scaling challenges of applying a product mindset across a large organization like RBC (22:02)
  • Insights into the framework for evaluating and prioritizing data product ideas based on their desirability, usability, feasibility, and viability. (26:30)
  • Measuring success and value in data product management (30:45)
  • Duncan explores process mapping challenges in banking (34:13)
  • Duncan shares creating specialized training for data product management at RBC (36:39)
  • Duncan offers advice and closing thoughts on data product management (41:38)

Quotes from Today’s Episode

  • “We think about data products as anything that solves a problem using data... it's helping someone do something they already do or want to do faster and better using data." - Duncan Milne (04:29)
  • “The transition to data product management involves overcoming initial resistance by demonstrating the tangible value of this approach." - Duncan Milne (08:38)
  • "You have to want to show up and do this kind of work [adopting a product mindset in data product management]…even if you do a product the right way, it doesn’t always work, right? The thing you make may not be desirable, it may not be as usable as it needs to be. It can be technically right and still fail. It’s not a guarantee, it’s just a better way of working.” - Brian T. O’Neill (15:03)
  • “[Product management]... it's like baking versus cooking. Baking is a science... cooking is much more flexible. It’s about... did we produce a benefit for users? Did we produce an economic benefit? ...It’s a multivariate problem... a lot of it is experimentation and figuring out what works." - Brian T. O'Neill (23:03)
  • "The easy thing to measure [in product management] is did you follow the process or not? That is not the point of product management at all. It's about delivering benefits to the stakeholders and to the customer." - Brian O'Neill (25:16)
  • “Data product is not something that is set in stone... You can leverage learnings from a more traditional product approach, but don’t be afraid to improvise." - Duncan Milne (41:38)
  • “Data products are fundamentally different from digital products, so even the traditional approach to product management in that space doesn’t necessarily work within the data products construct.” - Duncan Milne (41:55)
  • “There is no textbook for data product management; the field is still being developed…don’t be afraid to create your own answer if what exists out there doesn’t necessarily work within your context.”- Duncan Milne (42:17)

Links

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, I have Duncan Milne on the line, one of our Data Product Leadership Community members, and also a leader in data product management, and I want to hear more about this data investment side of your work as well, at RBC in Canada, correct?

Duncan: Yeah, that’s right.

Brian: You’re based in Canada, correct, too? You’re living there in—

Duncan: Yes, in Toronto.

Brian: Excellent. Excellent. Cool. Well, what does the director of data investment and product management do at RBC? Tell us a little bit about your work and your context?

Duncan: Sure. So, I work in the Chief Data Office, which is a horizontal function that supports our different lines of business: personal and commercial banking, so bank branches, credit cards, those sorts of things, capital markets, which deals in IPOs and institutional lending, wealth management, which is more, sort of, advisory for high net worth clients, and then insurance, are sort of our four main business lines. And the Chief Data Office supports those business lines realize their data ambitions.

Brian: Excellent, excellent. What are they ambitious about these days?

Duncan: Well, LLMs.

Brian: [laugh]. Of course.

Duncan: Yeah, that’s sort of all anyone wants to talk about.

Brian: Yeah.

Duncan: But my role specifically, so there’s two halves of it. So, there’s the data investment side, and what that is, it is a—I run a council of senior data leaders from all over the organization, and what we try and do is we try and define what the data ambitions are for the lines of business, and then make sure that our horizontal investment in infrastructure, our focus in specific use cases, match up with what the business wants to do so we can make sure that we’re supporting them adequately. And then the product management side—more relevant for this conversation—is around operating with a product mindset as it relates to the different data products that we create within the CDO.

Brian: Mm-hm, got it. So yeah, part of the reason we’re—not part of the reason; one of the main reasons here is you’re going to be doing a webinar coming up April 24th—have I got the date correct? I think that’s right—in the DPLC about how do we establish the value of one of these data products before we make it so we can figure out is it worth our time and resources and all of that. So, we’re going to kind of—if I understood, we’re going to, kind of, unpack a little bit of a preview, so to speak, of what we’re going to be talking about in the DPLC together. Is that more or less correct?

Duncan: Yeah, that’s right.

Brian: Excellent.

Duncan: And before we kind of get into it, I want to clarify that none of this stuff that I’m going to be talking about is finished product. It’s a work in progress. We are sort of finding our feet when it comes to working with data products, so I don’t want to position myself as the expert. These are just the ideas and how we are sort of tackling, applying this kind of nascent practice to what we do as an organization.

Brian: Frankly, I think that’s par for most organizations that are on this journey, particularly if they’re not a software company, and they’re not building, like, digital products as a business. That is very much a normal thing, that they’re on some continuum of this practice, staffing, skills, responsibilities, changing the organization culture to adopt a product operating model, or quote, “The product-y approach,” as I like to call it. Maybe for context, do you have a definition or—a short definition of what a data product is that you use inside the bank, just for our listeners’ context? Because we all know that there’s a lot of these different definitions floating around out there, and that might help us have some context.

Duncan: Yeah. Funny enough. Part of the reason why I joined the DPLC was to get that answer from you, but—

Brian: [laugh].

Duncan: I can take a crack at it, for sure, in terms of how we think about it. So, we think about data products as anything that solves a problem using data. That can be for an end client, or it can be for someone internally. In fact, a lot of the work that we do is more, kind of, internal-focused, enabling either our internal data teams or enabling financial advisors, branch employees, people in the call center, that kind of thing. But when we say, “Solving a problem,” I think what we’re really talking about is helping someone do something that they already do or want to do faster and better using data. So, there’s some business process that exists that needs to be carried out, and we’re going to figure out how we can apply advanced analytics and machine learning to help them either do it quicker or do it better.

Brian: I really like this because first of all the clarity on—and maybe you can unpack this—who owns the problem, and are your data product managers or whatever their titles are that are kind of leading this work, are they responsible for the benefit to the recipient? So, the user and/or stakeholders, if they’re not the same—you know, sometimes the person paying for it’s not the one actually using it, sometimes they are, do they own the response to the both the problem space and the benefit? They have to deliver a benefit, which is faster, easier, less frustration, more delight, whatever that may be, more clarity, more empowerment? Is that their purview to own that?

Duncan: Yeah. So, to an extent, I suppose, in the sense that, I mean, data science resources are expensive resources. Banks don’t exactly, you know, they’re not loose with the purse strings, so anything that is, sort of, a cost center needs to be able to kind of justify what they’re doing. And focusing on the benefit to either the stakeholder or the end-user is a great way to engender a positive relationship with those stakeholders, who have an option of whether they want to use us, or use another team within the organization, or just go it alone, use external partners like AWS or something like that. So, it’s a good way to—if you kind of focus on that, it means that we can not only deliver value for the organization, but we can also build those relationships that will help us build our future pipeline of work.

Brian: Got it, got it. Someone doesn’t just wake up and decide to do their work this way, in machine learning, analytics, AI, and I don’t know if this is something that you kind of are driving at RBC, this was already there when you got there, but what happened? What triggered this desire to apply a product-y method to doing this work?

Duncan: Yeah, this is sort of—the genesis of this work was a top-down push from our chief data officer, a fella by the name of Rex Davis. He came in, and coming from a more sort of retail-focused world, I think he probably has a customer in mind moreso than your traditional banking executive, which is typically more kind of product-driven. And it helped that we had a pretty strong product practice on the digital side within our personal and commercial banking group, routinely receiving international recognition for our digital products. So, Rex essentially said, “I want to do what they’re doing, but for data.”

Brian: When did that start? I mean, years ago? Months ago? Like, where are you, kind of, on that?

Duncan: About a year ago… yeah. We’ve been working on it for probably a year. It hasn’t been sort of—we had some growing pains at the start, just sort of achieving buy-in internally into why it was important to operate in this way, and we sort of just started to pick up momentum on that.

Brian: Got it, got it. You know, what I’m going to ask you now [laugh].

Duncan: [laugh].

Brian: So, Rex pushes down this desire to operate this way. I imagine you had to summarize that into a few short statements about, what does this mean at a high level in terms of how we do stuff around here? Was that clear, or did you have to go and, kind of, interpret that request and give some guidance to your team? Because I know you kind of manage this almost like—they’re almost like a group of data science consultants who are acting as data product managers: they’re kind of out on projects, and they help lead a line of business, and some initiative. And you can correct me if any of that’s wrong, but at some point, you had to give them some guidance or somewhere that that mission to go do it this way had to be communicated. I’m curious if there were a short few bullet points about what does it mean to do that? What are we actually changing around here?

Duncan: Mm-hm. Yeah, I mean, that’s been most of the, sort of, foundational work that I’ve been doing over the last year. When it sort of first got introduced, one of the things that I heard a lot when I talked about, you know, we need to be operating in this product construct was, “Oh, we already operate in Agile.” Yes, Agile can be a component of product management, but it’s not all that product management is. There’s a whole, you know, other component, or several components to it. So, the first step was clearing up what exactly we mean when we talk about product management. So, I did a lot of research and learning around that, and sort of tried to boil it down into three main pillars that function in kind of an iterative process. So, the first pillar is discover, where you’re, you know, doing the work to figure out who your customer is, what their needs are, why the problem that you’re trying to solve, et cetera. There’s the kind of build phase, which is more, sort of, the Agile-focused, you know, iterative delivery, sprints and things. And then there’s the measure pillar, which is about, okay, now that we have something out there, let’s measure it in an intelligent way that tells us whether what we designed answered the problem we’re trying to solve, and then iterate on that discover phase to figure out what the next thing we’re going to build as or amend within the product. So, there was that element of it. And then from there what I did—because a lot of people were at least partially functioning in that construct already, and it was different across the different teams that we have within the organization. So, what I had to do is I had to further define, so for each one of those pillars, what are the necessary elements, where if you have these things, you can say, “Yes, we’re operating within this product construct as it pertains to data products.” And then I had the team go through that exercise and score themselves, whether they had that element documented and updated, whether it was a more tacit thing where they thought they understood it, but they hadn’t officially created the documentation around it, or whether it was something new that they hadn’t done at all. So, I had them grade themselves in terms of what the critical elements of a product management practice is, and then that gave us sort of a baseline understanding so that we had specific things to work on, whether it was expanding our understanding of who the customers are and what their needs are, or making sure that we have appropriate metrics that are actually going to tell us whether we’re solving the problem we set out to solve. There’s a whole laundry list of things that it sort of gave us a baseline understanding, and then gave us a plan on what to work on going forward.

Brian: Got it. I want to get into some of the work that your staff is doing here, but I want to it—it sounded like even getting to the point of starting to do project work this way and product work this way had its growing pains, as you mentioned. Are you willing to share any of those? I’m sure this is like on a lot of minds of leaders that are in your seat right now that see the benefits here of operating this way, but putting it into practice may be challenging. Like, what are some of the roadblocks that you hit, or obstacles? Have you managed to clear those, or are you finding some success clearing them now? Do you think they’re just, everyone’s going to go through this because it’s a change? Maybe you could just tell us a little bit about that, if you don’t mind?

Duncan: Yeah, for sure. I mean, anytime you are going to someone and saying, “I’m going to tell you how to do your job,” you’re going to get pushback, especially when those people are working under the constraints of tight deadlines and on difficult work, they want to focus on the work specifically, and not necessarily take on a bunch of what they might view as paperwork or activity that is not focused specifically on delivering. So yeah, there was a good deal of pushback and stops and starts initially. What I decided to do to kind of overcome that was find the people who were interested in doing things this way, and use them as pilots. So, let’s prove out the value of what this does for you when operate in a product construct, and then utilize those folks as advocates for what you’re trying to do. So, you’re not just coming in and saying, “We’re tearing everything down. We’re going to do it this way.” It’s more. “Oh, you know, I found it really helpful to do this exercise to define this specific vision for this product because we had an understanding—or we thought we did—and then when we actually did this exercise, we found that different people were thinking about what the goal was differently. And going through the exercise of writing the product division was helpful in terms of making sure everybody was aligned.” So yeah, I think that we’ve overcome those initial growing pains of this is the way we do things because we’ve always done them this way, and we’re starting to gain traction and achieve some change throughout the organization.

Brian: Yeah. You made a good point here. I mean—and I think I have seen this with my own training—I think the most important thing, like, when we do skill development, it’s like—especially for adults—it’s like personal enrollment. Like, you have to give enough, frankly, not only just give enough, you have to want to show up and do this kind of work, which might be—it might not be writing code, building models, researching algorithms, reading white papers, it might not be that kind of work if you want to adopt this—in fact, it will have to have that kind of work. It literally can’t just be implementation work because you don’t know what you’re implementing, and you have no idea whether the implementation will create the benefit. I mean, even if you do product the right way, it doesn’t always work, right? The thing you make may not be desirable, it may not be as usable as it needs to be. It can be technically right and still fail. It’s not a guarantee, it’s just a better way of working. That’s how I see it. Can you rewire these people? Like, can you rewire these data scientists? Is that the approach, like, when there’s resistance? Or is it really, they have to either opt in or you really need to, like, change resources, keep that team on implementation because that’s all there—you can’t rewire that person, and find new talent that has this product management either direct past experience with it or an eager interest in learning it? How did you manage that, like, with the people that weren’t wired this way by default? Or are they still resisting? I don’t know [laugh].

Duncan: Yeah, I think it’s helpful to show the value where you can, like, impress upon them why it’s important in terms that they understand, and that’s not always super easy to do. Luckily, you’ve always got the—or at least I always had the fallback of, “Well, Rex said to do it this way, so we’re doing it this way.” But I think the more that you can empower them, make it less about you’re doing things this way from now on and more about, teach me about what it is that you’re doing, and I will help you put it in this framing, the more you can get people to talk about the value of actually gone through the process, the better. So, I don’t think that people can’t have their minds changed, and you just have to get new people in. You just have to find ways to highlight the value that they will respond to. One of the things that I found worked pretty well when talking about the specific sort of economic value of these data products, one of the things we got there was a lot of pushback around, we don’t know, it’s really hard to say what the value is at the outset, and if we put a number on paper, we’re going to be beholden to it, and then if we don’t deliver it, it’s going to be problems for us and everything. And I found that reframing that value assessment as a hypothesis, something to test against, rather than to be sort of like a set-in-stone statement of this is how much value we’re going to produce, kind of, helped people who are of an analytical mindset given to a more scientific approach understand what we were doing when we were trying to define the value at the outset.

Brian: Mm-hm. Do you think this idea of producing the benefit here, which is an ROI being a financial benefit, there’s other kinds of benefits as well, which might be making your financial advisors feel more empowered about which funds they recommend for new IRA accounts to college students—I don’t I’m riffing on something that maybe you might use an algorithm to do or something like this—making someone feel good and more empowered about their job, or helping the marketing department not waste money sending mailers out to the wrong people because they have no idea who to send their advertising to, or their, you know, their new product marketing stuff to. And it’s like, I’m going to look good to my boss if I’m not wasting millions of dollars sending out crappy advertising because we don’t really know who our targets are. I don’t think you can make someone care about making the marketing department happy because to me, the product thing is, like, you have to care about helping those people be successful, though in this case, they are—let’s call our users are the marketing department, the people that are doing advertising and mailers and all this, they probably have some mandate to spend efficiently and not waste money sending the wrong ads out. And I’m making up—I’m just imagining something that might be a use case in the bank, the data scientist, they have to care. I mean, a product person would naturally care about making those lives better because they know that will turn into business ROI for them. That’s kind of the game is the two-sided coin. Is this something where you have to kind of open the curtains up to your data scientists that are used to just working on the model to say, it’s actually more fun to produce something that someone uses, that is better for them, and it returns this value, which you can say my work created this fiscal value. I’m kind of riffing here, but I’m trying to understand. Can you convert someone to care about this stuff, or is it just, like, they’re wired that way? And tell me about that a little bit.

Duncan: Yeah. I think one of the things that I’ve found is that if a PhD data scientist has decided to go and work at a bank, they care that their work has application and value. If they only cared about the theoretical, they probably would have stayed in academia. So, I think that the more that you can empower them to use this highly specialized, really important skill that they have to have positive impacts on folks and communicate that it’s them helping to deliver this in sort of a, like, a self-serving way to get them to think about [laugh] it, to care about the impact, they will care about the impact. And that’s not to say that they don’t… like, their chief interest is building cool shit and working on the, you know, with the latest and greatest technology and everything like that, but you can get them to, I think at their core, they want their work to have impact, otherwise, they wouldn’t be in the field that they’re in.

Brian: Okay. I’m not sure that’s always been the case. I’m always curious if this is changing. And if someone asked me right now from all the people I’ve interviewed, I’d say, it does feel like it’s changing. It does feel like highly specialized, particularly in the data science field—my philosophy is Covid means lots more places to work, so it’s like, well, now what do you want? You can make money anywhere, but do you want your work to matter? You start to care about what the impact is, especially if you’re working at home all the time. It’s like, that’s my theory: people want to have an impact now; they want their work to matter. And so, it does feel like it’s changing whereas before, I definitely talked to people where it’s like, most of data scientists don’t care. They want to be able to write a paper about how accurate the model was that they built so they can get a citation or get, you know, some street cred with the data science community. That was more the interest than producing something that actually got used within their organization and created some kind of benefit. So, that’s good to hear that that’s actually happening. Can you tell us about any other of these obstacles that you’re either currently managing or that you’ve kind of—other strategies that you’ve used to overcome some of those obstacles you had implementing this?

Duncan: I suppose the next obstacle to overcome is figuring out how we appropriately scale the stuff because we do have quite a large organization, and depending on how familiar or self-starter-y the leaders are on the different teams, it can actually take a lot of focused attention to get them stood up and working on this stuff. So, just figuring out how we effectively enable the individuals who are working on these projects to take on this product mindset and execute is kind of a next problem that we’re working on.

Brian: Got it, got it.

Duncan: And if anyone has a solution for that, I’d love to hear it.

Brian: [laugh]. Well, there are frameworks and ideas, right, that we can use. It’s like baking versus cooking. And baking is very much in a science: the temperatures matter, the ingredients really matter, their proportions matter. Disaster awaits if you don’t do it, quote, “The right way.” I’m totally generalizing here. I don’t know if you cook, but cooking is much more flexible. It’s much more improvisatory. There’s general ideas about how to build a salad, and you want different textures. You want some sour, you want some sweet, you want some crunch, exactly how you put the salad together, it’s not like there’s a right salad and a wrong salad, and this kind of goes with cooking in general. So, I feel like a lot of this is more about the big ideas and making sure, like, when we step back, it’s like, did we produce a benefit for users? Did we produce an economic benefit? Can we measure that benefit? At the highest level, however you get to that, that’s effectively what we’re talking about doing here. And the culture, to me, is going to have a big bearing on how that’s done because it’s really hard to change the culture, especially if you’re new. So, it’s not like, just follow the rule book and then everything will, like, be perfect. I don’t think there’s a playbook to do it. I really think it depends on the maturity of the organization, how do they build software today, or digital solutions of any kind, whether it’s data products or not. It’s a multivariate problem, and I think a lot of it is experimentation and figuring out what works. And I mean, we saw this with Agile, right, which was, every time I go in, like, a client that [laugh]—“Yeah, we do Agile, but we have our own version of it.” It’s like, “I know you think you have your own version of it.” That’s actually the par for the course because everybody has their own version of it, so you’re actually not unique. It’s just, your own little version of it is unique, but no one is doing it by the book. Everyone is trying to follow some of the principles. And in fact, that’s a good example of literally trying to follow, like, a Scaled Agile Framework and some of these other things, it tried had to get so prescriptive, but what got lost with some of that is like, did we produce a benefit to users? Did we get a fiscal return on it? Is there any agility or are we just following a process now because that is the playbook that someone told us to, and it’s the easy thing to measure. The easy thing to measure is did you follow the process or not? That is not the point of product management at all. That completely has nothing to do with, like, delivering benefits to the stakeholders and to the customer. So, I think it can actually be taken too far if—you know, and this is maybe something, I don’t know, maybe analytical leaders need to really be cognizant of this is that it’s not—that there is improvisation that has to happen here. These are general concepts and frameworks, but no one is going to just give you a perfect playbook: just do this, you’ll get a 96.4 on the test, and now you’re doing it. Like—

Duncan: Mm-hm.

Brian: —it’ll never come across that way. It just won’t. It’s not that prescriptive because there’s too many variables. And anyhow, it’s not about me, I don’t want to get ratholed, but that’s my general take on the scaling, is it’s a lot of experimentation, being willing to turn off stuff that’s not working. It’s like you said: the hypothesis. I think that’s a great, especially for data type people, that’s a great way to do it. It’s like, “Here’s our current hypothesis for this quarter. We’re going to try this and then change it if it doesn’t work.” Now, it’s a learning thing. It’s like, oh, we all feel smarter now. I think that’s a great way to do it. But jumping back to your world here, the benefits and the value piece—so this is kind of getting into where your webinars going to go here—do you have, like, a framework for how you teach how to do this, like, with the data product leads, managers that are on your team? Like, how do you go from zero to one? Or how do you take the GenAI—like, what is our GenAI strategy, Duncan? What are we doing with that? There’s that solution in search of a problem, right? Like [laugh], how do we get to there’s a value, there’s a product vision, and now there’s delivery, and now there’s measurement? How does that start? You can use that example or pick another one if you want, but how do we do that?

Duncan: Yeah, so in terms of the data science work that we’re doing, I’ve recently introduced a framework with our teams to help them with the discovery process. So, when we do these one to two-week discovery sprints, it helps them frame up the problem in collaboration with the stakeholders so we can be clear on what we’re actually solving for on both sides of the fence, and then get into some of that definition of value. And I think that it’s important to understand that traditional economic value is only one part of the puzzle. It’s an important part of the puzzle when you think about what the business wants to achieve, they’re not just doing things to fill the hours, they want to increase revenue, decrease costs, and that’s kind of it. There are other things like decreasing risk or improving customer experience, but even those things are somewhat, at the end of the day, in service of those first two objectives. So, the framework that I introduced tries to take a broader definition of value, and I actually borrowed from Marty Cagan’s work, the four characteristics of a product being desirable, usable, feasible, and viable. So, when we talk about a product being desirable, you know, are we actually solving a problem? Are people going to want to use this thing? When we talk about it being usable, how does it integrate with the existing business process that we’re trying to affect? Is there a big change management ask associated with this? Do we have to develop a completely new process around it to make sure that it gets used? Feasible: does the data exist in a way that we can access it and work with it? I mean, as a bank, we have some real hurdles in terms of regulatory restrictions on sharing data and working with data in a cloud environment and all that, so are there any feasibility risks there that will prevent us from actually building the product that we want to build? And then the fourth one being viability. So, what is the economic value? And the team’s will go through this, sort of, four stage assessment within the discovery process, and the outcomes of that all get plugged into this model that are created to score and weight the different aspects of this. You can have a data product that has a billion dollars’ worth of potential economic value, but if we can’t actually access the data in order to build it, then it’s worthless. The same can be said for if we have this huge opportunity, but we are solving a problem no one asked to solve and there’s a perfectly acceptable alternative to the solution that we’re developing, so the product just never gets launched, never gets used. The billion dollars in economic benefit doesn’t really matter that much. So, we try and take the whole four aspects into account, but we try and weight them so that we are understanding that, at the end of the day, economic viability is sort of a critical component to what we do. And yeah, that’s the framework that I’ve introduced there. It seems to position things in a way that is valuable for both our data scientists and the stakeholders that they’re working with, and helps us with post-discovery go/no-go decision around whether this data product is going to provide value for the organization.

Brian: Got it, got it. You don’t have to tell me the specific number if you want to give a range, but do you feel like all this work you’ve done to date to implement this product-y approach to doing this work has shown some value, or at le—now you can actually show some fiscal value or other forms of value, and you could say, “Yeah you know, my team has created X,” or some—“We have an estimate. It’s this range of economic value that you have created with this model.”

Duncan: Not really because, as I said, we’re just in the process of changing the way that we do things. So, in the future, I feel like we’ll be able to say more definitively the value that we’ve produced, but now, in the current state, we really rely on the business to report back. Like, oh, here’s the value that was generated, and it’s not necessarily directly tied to—or cleanly tied to the work we do. It’s more of a value-supported approach.

Brian: How does Rex know if this is working or not? How do you know if it’s working or not? What’s your barometer for whether this product method is working?

Duncan: Right in the stage we’re at right now, the only barometer is how willing people are to implement it, how much pushing is required to get them to adopt these things. That is indicative that we’ve created it in a way that is valuable for the people who are going to be operating in this construct, and it’s understandable to them. As we move forward, there’ll be cleaner economic benefit that we can point to.

Brian: I’m curious where design fits into your discovery, development, and then the measure—I think those are the acronyms, roughly the three pillars?

Duncan: Yeah, discover, build, measure.

Brian: The DBM. I got—yeah, close [laugh]. But within that usability, utility, the customer willingness to use a solution, those are measurable design attributes. I’m wondering, do you do any type of measurement for that in terms of knowing, before all of the coding and all of the deployment and all of that is done, whether or not marketing is actually going to use this new tool to do their advertising, figuring out who to target their work to? Will they use this new decision support system—or whatever the output looks like, the format of that—how does that work? How do you come up with that to make sure that you don’t build the technically right, effectively wrong version of something?

Duncan: Mm-hm. So, within the desirability component of the discovery framework, we try and define, if we didn’t build this thing, what would happen? How would people do this thing that they’re trying to do if we don’t provide the solution? So, it gives us a sense of what the next best alternative is and what the delta is between what we’re trying to do and what that next best alternative is, and then presumably, if it’s significant, people will be incentivized to use our data products. And the other thing that we look at is the usability. So, we map out the business process. So, what are the activities that are in the business process? What are the people who are fulfilling those activities? And what are the technologies that they’re using? And if we’re not impacting that process in a way that makes it faster to do, or cuts out steps, or produces some change with limited change management overhead, then I think that people aren’t going to be incentivized, or it’ll be extra effort to use our data product. And the second you ask people to do more work to use your data products, you’re fighting an uphill battle, even if it is valuable.

Brian: Yeah, that takes a lot of research and intentional research to go even just map some of that stuff out. Who does that work? Are your DPMs doing that work? Are they the ones going out and figuring out okay, then what happens? Like, “Oh, so you call this other department? Who’s that? Who do you call?” “I call John and whatever.” “Okay, I guess I got to go talk to John.” And then you find out John then calls this other person and sends an email about whatever and uploads it to the tool. And it’s like, on and on and on. Is that your DPMs that are doing that work to map that stuff out? Particularly with complex banking, I would think these chains can get large.

Duncan: Yes, it is. You’re right that there are often real complexities in these business processes. In the discovery phase, it doesn’t necessarily benefit you to map out the specifically correct process. Simplicity is almost better, just so you can visualize what’s going to change in a way that people can understand. But once you get into the actual work, then you start to do your activity mapping and the more specific breakdown of the process.

Brian: Right, right. I’m more curious about who is doing that work, and particularly if it’s someone that’s done a lot of implementation-only work in the past, is that a hard thing to get them to go do that work, to care about it, to be curious about it, and see it as valuable? Because it is not building a model, it’s [laugh] not wrangling the data, it’s not data engineering, there’s no Python, it’s not doing technical implementation work. And I’m always curious, is that something you’re finding people are naturally like, “Oh, I see why it makes sense to do this. It’ll make all that implementation work a lot smoother. It’ll give me the information I need, and then I’ll know that it’s going to get used.” I don’t know if everyone connects those dots are not. Is that easy to get them to spend the time to do that and to want to do that kind of work, or is it kind of a push?

Duncan: Yeah, it’s definitely a push.

Brian: Okay.

Duncan: So, I’m usually involved in some of the support there to push them to do that, and help them work within the frameworks to define that stuff. Yeah, so when I talked about scaling, I can’t do that for everything that we work on, so it’s kind of slow-going at the start. The hope is that we can get to a position where our data product managers will, sort of, do that themselves. But it is a different skill set from what they’ve traditionally used, so that we have to build them up in that and give them the tools to succeed there.

Brian: Were there tools you used or trainings, or like, there was a zero to one here, right? And like, or whatever number you’re at, but there was the old state and there’s this current state. Is this in-house training that you’ve kind of developed, or has everyone gone through a boot camp? Or, like, tell me about how they got some of the skills to go start doing this?

Duncan: Yeah, so we have an internal team in our technology and operations that puts on product training. And we started sending people on that training, and one thing that we found was that, it was not data product specific, and it was primarily focused on direct consumer product, which as I said before, is not necessarily the kind of stuff that we spent the majority of our time doing. So, I’ve worked with the people who put on that training to develop a more specific class for our people, and others within the organization, to take that has that sort of internal data product focus to it.

Brian: Got it. The delta there was significant enough for you to feel like, hey, like, we actually need to get some data science and analytic specific knowledge transfer happening here. It’s not enough to just take regular product management?

Duncan: Mm-hm. Yeah, talking about things like a product maturity curve, looking at the more external aspects of product was less helpful. There’s still a lot of good stuff at that training, we’ve just sort of fine-tuned it to make it more impactful for data science practitioners within our organization.

Brian: Yeah, yeah. Not only do you have the technology, the solutions being driven by machine learning, and AI, and analytics, there’s also the multiple stakeholder situation because you have this idea of an internal customer, but then there’s a bank customer, and the internal stakeholder may not be the end-user. So, you have a kind of a complex ecosystem to manage to, at least that’s how I think about it, and I think that actually takes some extra work. It’s not as cut-and-dry as the commercial product management space where there’s a paying customer, and most product teams are not serving the business; they’re serving the external person that’s swiping a card or writing a check at the end of the day. So, it’s a little more complicated to manage all of that.

Duncan: Oh, yeah. Yeah, for sure, and it changes based on the specific business that we’re working with. So, for example, within our wealth management business, it’s a bit of a federated model where the wealth advisors, they own their client book and those relationships, and we try and enable them the best we can with data products, but they might view the access of their clients’ data for this purpose as sort of crossing the line in terms of the ownership. They certainly don’t want to be—like, the value that they provide the clients, they don’t want to be disintermediated from their clients by the organization that they belong to. So, it makes things on the wealth side a little bit more tricky than say on the personal banking side where those strong personal relationships don’t necessarily exist, and that’s more the traditional product setup.

Brian: Yeah, no, I can totally understand that. I mean, one of the things I always ask on engagements, you know, with clients is, “If this could go totally south, what would that look like? What concerns do you have about this? How could we royally screw this up?” And trying to get someone to open up about what their fears are with it, and suspicions that can help us know how to, “Oh, I didn’t even need to know. I mentioned, we’re going to use synthetic data. We’re going to actually create all this fake data based on your real stuff, and then we’ll never touch that again.” Like, “Oh, I didn’t know that. That makes me feel better.” You know [laugh], or whatever it may be. But you don’t even know to say that because you don’t know what their concerns are, and so giving a space for stakeholders to talk about their fears—and that comes with having a relationship, which means you have to be talking to these people and getting to know them, otherwise they’re not going to be vulnerable with you, and they’re not going to share some of that stuff because it may take a little risk for them to open up about some of that. But it sounds like you’re doing some great work transitioning to this model, and I appreciate you being so transparent here about what’s going on behind the scenes there. Not everyone is comfortable doing that, so this has been really a great conversation, so thank you.

Duncan: Great. I’m glad.

Brian: Yeah. Do you have any final thoughts? I mean, you’re going to—I know we’re going to go deeper for members of the community. You need to apply to be a member of the Data Product Leadership Community, but Duncan’s going to be doing a session on this, and then we’ll have a group discussion about how we establish the value of data products before we make them, on April 24th. It’s at 12pm Eastern. Is there anything else, Duncan, you’d like to add, either about your session that you’re going to do, or just closing thoughts or advice for leaders that are in your position right now, changing to this model?

Duncan: Mm-hm. I would say that one of the major discoveries as I’ve been working through the initial phases of this work is that data product is not something that is set in stone. Data products are fundamentally different from digital products, so even the traditional approach to product management in that space doesn’t necessarily work within the data products construct. So, for other professionals in this space, I would say, don’t worry about doing things by the book. There is no textbook for data product management; the field is still being developed. You can leverage learnings from a more traditional product approach, but you used the analogy of cooking; I would use the analogy of jazz, where there’s a framework, but there’s a lot of opportunity for improvisation within that framework. And the one piece of advice is, don’t be afraid to improvise. Don’t be afraid to create your own answer if what exists out there doesn’t necessarily work within your context.

Brian: I love that you brought this to jazz at the end, since I’m also a jazz musician. I did not want to use that analogy because I [laugh] felt like people wouldn’t understand it, but the fact you did is fantastic.

Duncan: Well, maybe we can, once we stop recording, we can talk a little bit more about jazz.

Brian: Okay [laugh]. Well, Duncan, thanks. This has been really great. Duncan Milne from RBC up in Canada. Thank you so much for coming on and being so transparent and open about what’s happening behind the scenes.

Duncan: Yeah, it was my pleasure. This was fun.

Brian: Cool. Thank you. Take care.

Array
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Subscribe for Podcast Updates

Join my DFA Insights mailing list to get weekly insights on creating human-centered data products, special offers on my training courses and seminars, and one-page briefs about each new episode of #ExperiencingData.