Today I sit down with Vijay Yadav, head of the data science team at Merck Manufacturing Division. Vijay begins by relating his own path to adopting a data product and UX-driven approach to applied data science, andour chat quickly turns to the ever-present challenge of user adoption. Vijay discusses his process of designing data products with customers, as well as the impact that building user trust has on delivering business value. We go on to talk about what metrics can be used to quantify adoption and downstream value, and then Vijay discusses the financial impact he has seen at Merck using this user-oriented perspective. While we didn’t see eye to eye on everything, Vijay was able to show how focusing on the last mile UX has had a multi-million dollar impact on Merck. The conversation concludes with Vijay’s words of advice for other data science directors looking to get started with a design and user-centered approach to building data products that achieve adoption and have measurable impact.

In our chat, we covered Vijay’s design process, metrics, business value, and more: 

• Vijay shares how he came to approach data science with a data product management approach and how UX fits in (1:52)
• We discuss overcoming the challenge of user adoption by understanding user thinking and behavior (6:00)
• We talk about the potential problems and solutions when users self-diagnose their technology needs (10:23)
• Vijay delves into what his process of designing with a customer looks like (17:36)
• We discuss the impact “solving on the human level” has on delivering real world benefits and building user trust (21:57)
• Vijay talks about measuring user adoption and quantifying downstream value—and Brian discusses his concerns about tool usage metrics as means of doing this (25:35)
• Brian and Vijay discuss the multi-million dollar financial and business impact Vijay has seen at Merck using a more UX  driven approach to data product development (31:45)
• Vijay shares insight on what steps a head of data science  might wish to take to get started implementing a data product and UX approach to creating ML and analytics applications that actually get used  (36:46)

Quotes from Today’s Episode

• “They will adopt your solution if you are giving them everything they need so they don’t have to go look for a workaround.” - Vijay (4:22)
• “It’s really important that you not only capture the requirements, you capture the thinking of the user, how the user will behave if they see a certain way, how they will navigate, things of that nature.” - Vijay (7:48)
• “When you’re developing a data product, you want to be making sure that you’re taking the holistic view of the problem that can be solved, and the different group of people that we need to address. And, you engage them, right?” - Vijay (8:52)
• “When you’re designing in low fidelity, it allows you to design with users because you don’t spend all this time building the wrong thing upfront, at which point it’s really expensive in time and money to go and change it.” - Brian (17:11)
• "People are the ones who make things happen, right? You have all the technology, everything else looks good, you have the data, but the people are the ones who are going to make things happen.” - Vijay (38:47)
• “You want to make sure that you [have] a strong team and motivated team to deliver. And the human spirit is something, you cannot believe how stretchable it is. If the people are motivated, [and even if] you have less resources and less technology, they will still achieve [your goals].” - Vijay (42:41)
• “You’re trying to minimize any type of imposition on [the user], and make it obvious why your data product  is better—without disruption. That’s really the key to the adoption piece: showing how it is going to be better for them in a way they can feel and perceive. Because if they don’t feel it, then it’s just another hoop to jump through, right?” - Brian (43:56)

Resources and Links:

• LinkedIn: https://www.linkedin.com/in/vijyadav/

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill and I’ve got Vijay Yadav on the line today from Merck, specifically the manufacturing division. And you’re the head of data science in that group. Is that correct?

Vijay: Thank you, Brian. Yes, indeed, I’m heading data science team at the Merck manufacturing.

Brian: Yeah, yeah. So, you had reached out to me and we’ve been talking a little bit, and you’re doing this whole—you know, it’s going around since, like, the Covid, or something—this product-driven orientation centered around user experience and customer and user, the human-in-the-loop piece, when we’re talking about building enterprise data products, specifically with machine learning and analytics. And I thought it would be great to have you on the show to talk about some of the methods you’re using, what are some of the results that you’re getting from doing things this way?

So, tell me though first, like, why do you even care or know about product managers and user experience in the realm of data science? Because this is still somewhat of a new thing. Did you always do things this way or where’s there a moment in time where you switched and said, you know, we’re going to borrow some stuff from, you know, software development methodologies and apply that to data science work? How did that happen? Was it a change? Like, give me some of the backstory there?

Vijay: Yes, sir. Thank you, Brian. So indeed, the data product management concept, I brought in. I have an experience prior to joining Merck, so this has been with me for some time. And I think what I have found in my experience in applying that concept, is the right way to do, and I think the whole adoption element becomes much more [unintelligible 00:02:14]. And we’ll talk more about that.

So, let me share my experience. What is the driver behind the data product management? So, think about anytime we are delivering a solution. There are two levels that we want to look at. One is the business level: why business will take your solution right? And that is basically about the outcome, the business outcome that solution is providing.

So, whatever the solution that is being developed as data product, the very first assumption—and that the requirement is that it is really solving the real problem and given the outcome you’re looking for. But in order to be adopted by any business or any company, let’s understand who are the people who are going to use that solution. Now, let’s talk at that level, at the customer level or user level, right? Why should they use this solution? What is wrong with the solution they have been using currently in the process, right?

So, what are we basically giving to them, to the solution? Fundamentally, there are a couple of things that is really key. Number one is experience. Are we giving, the solution is giving a different experience than what they have today? So, when you talk about the experience, it is in terms of the speed.

So, think about today, any decision-making process. Consumer is taking the data, making certain decisions so that they can achieve the outcome they’re looking for, right? So, the one, number one piece is the speed. So, if currently the user is whatever the time they’re taking, if the solution is giving much faster speed, yeah, that is a fact that we want to consider, right? So, that’s part of the one component.

Reliability, right? So, how reliable is the solution that is being given for them to make the decision-making process compared to the what they have today, right? So, the reliability, trustworthiness, all of those elements, and also, you know, one of the things, you know, Brian, I see that they will adopt your solution if you are giving them everything they need so they don’t have to go look for workaround. So, think about an existing solution where someone is using Excel sheet, whatever the other solution they’re using it, and you provide them a solution who does half of the things what they were supposed to do, right? While using this new solution, if they have to go back and do some workaround and do half of the thing, you know for sure they are now going to adopt a solution.

So, the idea here is that data product gives the best experience possible to the user so they can adopt the solution with the interest and build upon it, right? So, there’s a whole lifecycle of building the solution, in a very true north sense. So, I think those are some of the elements we want to consider for bringing in the data product into the mix while delivering the solution.

Brian: So, would you say it’s true that the primary reason you do things this way is to increase adoption? Is that the problem you try to solve with this methodology is raising the adoption of the usage of these tools that we’re creating? Is that the primary driver, such that you can then get the business value?

Vijay: Yeah, absolutely. So, think about, right, so we made two assumptions: that solution is going to deliver the business value, meaning it is giving some insights to make a decision to get the outcome; but at the very user level, right, if the user is not going to adopt it, then the first outcome you’re looking for, it’s not going to be there, right? So, I think it is—so that absolutely is definitely a big factor in into this element of having a design thinking into the data product that we have.

Brian: Yeah. It’s hard for me because I work in this space to imagine why there would be a reason not to do things this way. But there are—you know, this issue with adoption is a problem in a lot of organizations. Like, do you know why that is? And what’s the resistance to changing it to doing a methodology more like you’re talking about, which is very much rooted in, we have to make someone’s life better.

If we don’t make that—like, in your case, someone on the manufacturing lines, if we don’t make their work easier, faster, better, some improvement in their life, there’s no way you’re going to get any business return on investment, right, for the expensive data, the modeling work that you did, and all the technical stuff that happens in the background, that’s just a giant cost sink, right? Why would someone do it that way, like, what are the issues in other organizations that you’ve worked with? Or maybe you’ve had the fortune of not seen those, but—

Vijay: [laugh].

Brian: It’s a problem in a lot of other places?

Vijay: Yes, indeed, is the problem. So, think about, you know, data and analytics, and data product concept, as you said, is maturing, right, this particular concept. Traditionally, what we have seen in the past is that, you know, if the system needs to be developed, you go to the user, you take the business analyst, and they go back and talk to the users and they define the requirements. And user tells them, “Hey, this is what I’m looking for.” You go away, you come back, and you give a solution at the end.

And user said, “Yeah, I said, exactly, at that time, but that’s not what I meant,” right?

Brian: Yeah.

Vijay: So, now you are at the very end up of thing, right? Now, the experience that user expecting somehow would be never got out of their mind. And translated there’s a thinking, right? So, the aspect of the thinking, how the user is thinking, what experience they want, I think it’s very difficult to capture in the traditional user requirement, aspect. And traditionally, that is how everybody has been doing, right?

So, it’s really important is that not only you capture the requirements, you capture the thinking of the user, how the user will behave if they see a certain way, how they will navigate, things of that nature. I think those are some of the key factors that makes us ready to get into the data product where you bring the user thinking into the aspect of that. In the past, if we follow a very traditional methods, what happens in the end? User said, “Well, I am probably this is not what I give.” But you know, what happens. You deliver the solution, you know, team goes away, and you know there is no adoption because the experience they were looking for was never there.

So, it is a traditional method of developing a solution in older days. Nowadays, you engage the user. They are part of the journey of developing the tool. That’s a whole data product management is all about, that you do the user, you know, research, you talk to the user and think about a business problem that is never happens in isolation. Somebody upstream causing the problem; somebody downstream getting impacted.

So, when you’re developing a data product, you want to be making sure that you’re taking the holistic view of the problem that we can solve and the different group of people that we need to address their need, and you engage them, right? So, to me, when I am trying to solve developer data product, I say, “Okay, business user, you frame your question. What question you want me to answer through the data?” You frame it, and the next question from me would be, “You tell me how would you like the answer to be look like?” Let them describe that. “I want this way. Give me a dashboard or give me alerts, or give me a [unintelligible 00:09:28] board.” Right?

I think that’s how you want the experience to come into discussion, and you want to capture those kinds of thinking into your, you know, the prototyping and, you know, low-fidelity, high-fidelity, whatever, you know, method we’re going to follow to capture those. Now, if they’re part of the journey and the whole development process, that is one we know at the end it’s going to be adopted, right? So, I think there are some traditional way of doing and this is getting the product management, there’s a quite a bit of difference. And you would think it’s so obvious; why we don’t do it? Unfortunate, that’s the reality. It’s not being used. And I think, as you [unintelligible 00:10:08] practice on myself, I have seen the success really well and I always try to adopt, even within the team I promote that, the data product management concept in developing the solutions.

Brian: Yeah, yeah. You said a couple of good things, I wanted to challenge you on one thing. So, I’m in agreement, you know, you can’t—in my experience, that this whole analyst go and get a list of requirements, take it back, deliver it to the people who execute on it, it feels really good because you got those lists. Like, here are the instructions on what I want you to make for me. Check everything off, and then you’ll be done.

And it’s like, “Great, I don’t have to think I just have to, like, go execute.” The issue with that is that a list of nouns is not the same thing as an experience. And most people aren’t going to list out an experience for you because users are not designers. Yeah, users don’t—can’t—most people cannot objectively look at their workflow, they don’t think about at a 10,000 foot view, “Here’s how I make a decision on pricing widgets for the next, you know, two quarters,” or, “Here’s how I do some audit, whatever.” Like, they’re not thinking of it that way. They’re just going step by step the way they always do it.

And so, that type of observation about how is it now, how much better could it be, where are the friction points, it’s very hard for a user to step out of their skin and say that, so I think you have to be very cautious about getting a customer or user handing you a list of things to go make and assuming that if you just do that, you’re going to have a great result in the end. So, the thing I wanted to push back or ask you about is even when you ask them, how do you want to see it? To me that’s—while there can be some validity and asking that question as a kind of a beginning of a conversation, like, “Well, I want to see it in Tableau.” My challenge with that is like that may be there because that’s the only tool they’ve ever seen a chart in and they actually think they need a chart, so they give you the tool that’s the only one that they know how, even though, let’s say, well, yeah, but this is a traveling salesperson and Tableau doesn’t work great on the phone. But they’re telling you that because whenever they are on their laptop, they use Tableau, so they say I would like it in Tableau because they can’t imagine that, oh my gosh, this could be in a mobile application, or it could be an email sent to me, I don’t actually even need to log into a tool.

Talk to me a little bit about how much you accept that requirement that they give you in terms of how the interface should look or the user experience. I can see issues with that self-diagnosis going wrong, or at least not so much wrong, but the users can’t imagine how it could be better than the thing that they know what to tell you. Because that’s what they know. I only know Excel. So, I would like to get it in Excel. Well, of course you would because you use that tool all the time. But maybe there’s actually a better way, you just don’t know it. And that’s the role of design and product. So, talk to me about that a little bit, like, asking them what delivery format they want to see it in—

Vijay: Yeah.

Brian: —and that dance.

Vijay: Sure. So sorry, I didn’t clarify that. Technology is never part of their ask. When I asking a business question, forget what technology. We’re not talking about whether we’ll use Tableau, Excel, that is not part of the question.

What I meant was by asking them more like user stories: tell me what you want. The second element of that would be, why you want, right? What is the rationale behind it, right? And who you are as a user. So, your role, what you want, and why you want, right?

This could be at a very high level. Forget about technology. So, somebody says, “As a, you know, technical lead on the shop floor, I want a decision to be made about when to go and”—let’s say I’m making up—“Take a decision for rejecting a batch. So that, you know, we can avoid, you know, further, you know, waste of time,” or something like that. It may not be a good example, but the point I’m trying to make is technology is not part of the solution.

You tell me what you want. Now, let’s dive down into that exactly. So, if you want something like that, what do you want to look like? Let’s frame it more like a wireframe, put them in a in a diagram, you know, low fidelity, kind of wireframing, things of that nature. You start jotting down a little bit more and more than this.

And also what’s important for them? So, when you get an answer, answer might have ten piece of information. Which one is more significant for them, right? And you keep that thing at the very top. So, you prioritize what is important for them, giving them a bigger answer, right? If the answer is big, then what do they want to see, right?

So, I think I would not go with them, you know, discussing what technology and tools you’re using. That irrespective; that’s agnostic. I think what I meant by that is that so how do you want the answer to look like at the user interface and experience you do that? Also navigate, right? So, how do you want the thing to flow?

So, if you’re moving to the screen, right, how do you want to see what you want to see the first and second and third? And you go, there’s a whole iterative process. I don’t think you can get everything in just one go, right? But the idea is that engage the user in defining the user interface and also experience, right? Fast. What do you exactly want? Give me one [unintelligible 00:15:31] relevant to me.

So, anytime we design a system, if I am a [unintelligible 00:15:36], I’m not interested a lot, many other things. Give me exactly what is relevant for me, right? So, as a system designer, you want to filter everything out, give me exactly what I’m looking for, right? So, capture those kinds of elements in part of the designing of the system, so that they can basically get a better experience and to adopt the solution.

Brian: Yeah. Just to clarify my question, I was actually talking about not so much that you’re bringing in an implementation idea, but more they’re requesting an implementation because that’s what they think they know. Like, “I would like a machine learning model that will do X,” that’s a loaded requirement statement, right?

Vijay: Right.

Brian: It assumes that you need machine learning at all. Or, “I would like a Tableau dashboard that will help me on the manufacturing floor.” That’s a bunch of nouns; it presumes a technology; it has no problem statement built into it; it’s a tactic. And so, it may or may not be correct, but it was more of that when they—you know, and maybe this doesn’t happen, but this is the kind of thing in design that we’re always trying to look at objectively, make sure we’ve properly done the research and dug into why someone thinks they want that. And sometimes it’s correct or it is the best logical path, but it’s not always the best way to go do it, right? Like sometimes that can change.

And maybe you’re saying you find out through the low fidelity prototyping work, whether it’s right. And that’s what I really do like about what you’re doing is—you know, I just heard another designer frame it this way: “We’re not designing products for them. We’re designing products with them.” And I think that’s a great framing. And when you’re using low fidelity, it allows you to do it with them because you don’t spend all this time building the wrong thing upfront. And then it’s really expensive time and money wise to go and change it.

You’re designing it with them. And I don’t know, maybe that’s how you arrive at a different destination, potentially, even when they said I think I want to go to Austin, we arrived in Los Angeles, but everyone’s actually happy. You know.

Vijay: [laugh].

Brian: You know? Like—[laugh]. Talk to me a little bit about that. Like—

Vijay: Yeah. No, I think that’s a good point. And I’ll give a couple of examples where when we talked to the user, and user states the problem: “This is what I want.” And I asked the question, “Okay, tell me, if you are the king, what do you want your solution to look like?” And the guy said, “I don’t know.”

So, think about their point of view. They can illustrate the problem but they don’t know what the answer looks like. That is a very weird, how do you know the problem if you don’t know what the good looks like? Right? So, I think for them, it takes a couple of iterative process where I think I understand what they’re saying they’re not able to convey, and then I go back and create the low fidelity, kind of, prototype myself, and next day, I go and say, “Is this what you mean?” I said, “Yeah, that’s what I mean.” Right?

So, I mean, things of that nature, sometimes user is not able to tell you what exactly the one without what looks like. So, that’s again—

Brian: “I know it when I see it.”

Vijay: [laugh]. That’s right.

Brian: [laugh].

Vijay: That’s right. That’s [crosstalk 00:18:31]—

Brian: Right?

Vijay: Exactly that’s right. So, I think that’s the reason that you made a very good point. You want to design the solution with them, not for them. I mean, that’s really key.

Brian: Yeah. Yeah, yeah. And you know, when I hear that, too, it’s like, “Good, then that’s exactly what we’re going to do because you’re telling me, I got to wait until I can see it. Great. That’s what we’re here to do is to actually help you do that.”

Because everybody has an opinion about design, but getting from the zero to the one of design is often a lot harder than getting from the one to the two. So, doing this work of interviewing users, understanding what status quo looks like, understanding, like, okay, if we’re at a 7 and we’re trying to get a high score, what is a 20? Like, what does it mean to jump up more than a hundred percent, to get twice as good? Define what that means, in words for now, through conversation, and then go and translate that into an experience, and then you do that integration to figure out are we actually getting towards that 20 marker that we’re shooting for or not? And a lot of people can’t get to the—the zero to one part is the hard lift, but it gets a lot easier if you really understand the problem space really well. That journey is usually a lot easier. I don’t know if that’s your experience, too.

Vijay: 100% agree. I mean, that’s exactly right. So, the beginning is definitely the start, you know, the difficult time, right? Once you get into that, I think it’s a much, much—becomes much easier, possibility to do that. I agree with that.

Brian: Yeah. So, tell me in general, like, do you feel like user adoption of data products, at least in the manufacturing division, is that not really a challenge for you? Is this an ongoing challenge? Like, how was that challenge for you guys at all, if at all? Like, how would you—on a scale of one to ten?

Vijay: No, I think adoption is always a challenge, right? This is not one thing that we’ve seen, it’s very common almost everywhere. So, think about the human being. They have been working in certain ways for last 10, 20, 15 years. Why do you think that they will change overnight to do that, right?

So, I think adoption, if they’re used to it, even they’re spending a lot of time, so think about someone who’s spending, you know, two hours, three hours a day doing some non-value-add using Excel [sheet 00:20:43]; you’re giving a solution for them to do in five minutes. Some there are people—you know, and usually the human nature to adjust the chain order[unintelligible 00:20:51], three hours is fine. I’m used to it. I can do in my sleep. It’s X of C. I just put a number here and there.

Yeah, 15 minutes. Yeah, I could go and do some extra work. And, you know, I’m happy where I am, right? So, I think there’s a whole shift of the mind thinking, and especially in a data culture, right? So, data is powerful. I can tell you that in normal situations, not everybody sees the data value in the same way.

And there is whole gradual progress and in how someone sees the value of the data success. I have seen the users who have never kind of understood what data can do for them, but once they got a small piece, and they say, “Oh, my goodness. How about this?” And you see that’s where we’re starting with quantities. So, you said the very beginning piece is the major piece. Once a user is in that dialog the data can do for them, then they are there basically to go to the level that, you know, even where the product leaders—our data science leaders—we never imagine where the user can take you.

Brian: I think I’d follow on to what you just said. It’s also really important. If you’re in this product management role, even if you’re an internal enterprise organization, part of that lens, I think, is understanding does this person actually want this help, too. It’s not just that, like, “Wow, machine learning would be a great application here. We could blow this roof off this place with business value.”

Yes, technically, that might be true and maybe you have all the ingredients to bake an amazing, predictive model and all this kind of stuff, but if that person doesn’t want it and they don’t want to make the change, it doesn’t matter how ripe of a data science or analytics problem it is, right? You have to make an impact in their life on a problem that they care about, right? Like, “I dread, I hate using the spreadsheet and doing four hours a month. Oh God, the end of the month cycle, like, it kills me.” Those are the kinds of things where you’re probably going to hit a home run even if the technology piece was small.

And tell me if I’m wrong, but if you solve their headache, that four hour headache they have every month, not only do they get a taste of what data can do for them, but maybe they’re getting hooked a little bit now on, well, where else can this help me out? You’re starting to build trust there, and you’re creating an impact because you change someone’s life. Even though the business got some value, which is, like, “John’s not wasting four hours a month, like, punching numbers into a spreadsheet. Now, his brain can be doing more strategic work.” And then there’s some downstream business value. And the data is accurate. And there’s less guessing, which means the predictions are all better.

Like, you can roll it downstream and see the outcomes, but it started with making that impact in someone’s life. Do you see that as well where, like, solving it at a human level is kind of that first step to getting the business value piece?

Vijay: Oh, that is such a great point, Brian. Absolutely. I think so—

Brian: Yeah.

Vijay: You know, one of the element of making any data product successful is the change management, right? So, when we talk about change management, it’s about three elements of that, right? So, what are three things you’re changing? Number one, you’re changing the process, the way they have been working, right? Then you’re now going to have a new process. Technology might be a different element. And the most critical element is the people side, the human side, the change in the human side, right?

Why should somebody see—can they see the value in there? Can you make them buy in? If you develop a solution and they have not bought into that? I guarantee you they’re not going to adopt it. So, I think how do we manage the change to make them buy in?

And I think that is the very critical pieces. Can you make them, right? So, getting the buy in? But other thing is do they have the knowledge and capability to make that happen as well? So, do they have a confidence? So, somebody can say, “Listen, yeah, it’s a good tool, but I’m not good at technology. I don’t know how to handle, you know, the computer.” Right?

So, given the solution, they like the idea, but they don’t have capability to use that idea to get the value of that, right? So, I think these are some of the elements of change management. So then, now, you have to talk about changing the culture of the people. So, make them data literacy program, how to handle computers, for example, right? So, now you have to move that piece as part of the change process, in order to make that adoption, you know, at the next level, if you think about.

That is a whole gradual progression, right? So, if you think about, you know, 60,000 people, right, not everybody has the same level, right? Literacy programs and the whole data culture element that basically comes as part of the data change management process overall.

Brian: Tell me, how do you count—what are the metrics for adoption? Like, we’re talking about—we said that this model increases it, so it suggests that we had some baseline and we know it increased? Like, how do you count that adoption went up? Like what are your metrics for measuring adoption and then downstream value? And can you tell us what some of the—what is the value that this process has created? Like, can you quantify any of that for us what this what this method is brought to Merck and your previous work?

Vijay: In terms of the metrics, one is the very true north which is the business outcome? So, think about we are solving a problem, let’s say in this one solution that I’m referring to is, it’s basically creating the insights that can help the people at the shop floor to improve the yield, right, and reduce the false rejects, for example, right? So, if the insights are getting created and you establish the baseline, and you’ll see that just by the system being used and you’re able to deliver the yield that you’re looking for, somebody is using the system because without use, you won’t be able to do that, right? But that’s a true north. Prior to that, I think that option comes in: let’s understand how the system is being used.

Can we track the log of who’s going and doing what? How many reports have you created, right? For example, in our case, in certain cases that we’ve seen is a number of analysis that has been created the last six months, right? So, if you’re analyzing the data, you’re creating the [unintelligible 00:27:03] analysis, you’re addressing the issues. And on production floor, you know, there’s always some useful that you have to that.

So, we’re maintaining the log, how many issues we are addressing using this specifically tool, right? So, those are some of the factor, how much time has been spent by the users, we can basically track that as well. So, the people are spending time on your system, you know they’re not sitting idle, they’re doing something, right? So, the time used and how many reports that have been created, right, there’s a whole number of problems that being solved there are use cases, basically, that has been identified that this system is being used for multiple use cases. Let’s understand how many use cases we have solved this, and how we are basically applying.

So, there are multiple factors in terms of the users time, the number of people using it, how many use cases have been kind of—this tool had been used for? And the very true north is what business outcome has been basically delivered?

Brian: Got it. Got it. Do you do any testing of the users using the live systems at all to see if the tools once, in production, are actually as easy and valuable as we thought? Because the metrics of, like, counting reports and looking at logs statistics and all that, it is an indicator, but for example, time spent using the tool can also be a tax. It can be, “Wow, they’re spending nine hours a week on this?” Like, that was not what we—we thought in 15 minutes, like, a day. That should be plenty. What are they doing spending nine hours?

Like, we’ve actually introduced a tax here. So, how do you know that the time is well spent and that they’re actually getting the value out? Like is that—do you just count, like, for example, like, “Did the yield on the floor go up?” Like, we have 10%, less rejects, and it correlates with the date we went into production, so we kind of associate that business metric with the launch of this tool? Is that kind of how you do that, or—

Vijay: That’s definitely one of my [picks 00:28:59] at the end of the day. What the business cares is, you know, whatever the time you spend in the system, tell me what you deliver for the, [laugh] you know, the outcome. That’s how basically, you measure. But in terms of the adoption, what I meant was not necessarily, I would say the number of user is really important. So, think about what is the target audience. What is the number of customers, and how many people are going there?

So, let’s say we designed a system that we know this system would be used by, you know, 100 people, okay, average, let’s track down how many people on a given day they’re using it, right? Average time is spent on that. So, not only the time is spent, but how many number of people that were basically using it, right? We’re not sophisticated to the level that where you do the web analytics, where Google and other people are really going and tracking and, you know, they’re tracking every move, right? So, we are not in a situation where you’re tracking the funnel, for example.

So, if the user goes on the website, try to buy a product, right, and you see where does the user drop, right? They were searching and they clicked the button, they put in a shopping cart, and that’s where they basically dropped, right? We’re not—in this particular case, we’re now on the manufacturing side, we’re not using tracking to the level, “Okay? Where did you go in the system, you dropped it?” Ideally, someday, I think we might be able to do it, but I think our metrics is a little more higher level than that.

Brian: Got it, got it. It just I want to throw in that there are qualitative and quantitative methods of doing this, but actually doing ride along interviews, where you actually in this case, like, this is a very common thing that a user experience professional would do would be to go and watch someone on the shop floor—like, at Merck, for example—watching them use this tool in the context of their real job, very much an ethnographic kind of approach there. Another way is to actually give them tasks in a lab setting where you give them realistic tasks, like, your job is to make sure the number of, you know, pill bottles going out the door is as high as possible. And if the system is suggests a reject, it’s your job to choose whether or not actually it should be rejected or not. And you give them some tasks there in the tool and you watch how they go do that.

And you figure out where are they spending too much time too much friction. You look for places where the design is failing them, so that you can then switch that. That can then give you that good benchmark for, like, okay, on average, people should be able to do this in about seven minutes. So, if we look at our quant stats now, we see some, like, people that are spending hours, on a high level, it says something might be wrong here, like, what’s going on? Why are these people spending so much more time?

You know, are they doing research? Are they running custom reports? Like, what are they doing? And it gives us some signal about where we might want to go continue our research work. But I don’t want to get too much on a tangent with all that.

I wanted to jump to some of the financial or business impact that you’ve seen with this. You know, I understand you can only share what you can share, but even on order of magnitude, like, how has this approach shown its value in terms of the business impact? Is there anything you can quantify about how—you know, we built this data product for the manufacturing floor? And you know, before it was like this, and after, it’s like that? Can you tell us a little bit about how any impact that you can actually measure?

Vijay: Yeah. So, Brian, that depends on the data products. We are just—I’m am part of the multiple data products, but one use case where we developed a data product where it is providing the insights for shop floor team members as to if the product getting rejected from the products on line, and some of them are good product, right? It is giving them insight as to what is happening and what kind of defects are these. Now, those insights are very [unintelligible 00:32:46] and, you know, the decision maker can take that information and understand as to what is happening in the system.

So, for example, if they see that there’s a lot of [unintelligible 00:32:56] defects coming in, right, in the process in this particular batch, they can easily go back and see, you know what? [unintelligible 00:33:05] happens because when we sterilize the, you know, vial and syringe, you wash them and you freeze them. Now, if there’s a moisture left when the washing process before you go to the freezer, those things are disturbed, all of a sudden, basically you see assembly line, the [unintelligible 00:33:23]. They can go up in [unintelligible 00:33:24] process, tell them the next batch you run it, your strategies and process and feeding process was not correctly done. Once you do that, you’re not going to be getting that same insights, right?

So, there are different types of defects. So, you’re not—and these are some of the insights that you can never get easy. So, we’re talking about so many images. [unintelligible 00:33:45] it is not possible at the end of the batch which images are defective, what kind of defect it is. It’s almost impossible. I don’t have time to wait for the next batch, right? You got to finish fairly quickly, right? There’s no time for that, right?

So, you want a tool that can give you the insight before you launch the next batch, what are the insights can we create so that we can adopt the change, right? So, that’s one example. Now, if we are able to use this tool to improve the yield, you have already invested all the raw material, labor, machinery cost, all the time had been basically spent. So, all the time is spent, so what we’re trying to do is we want to capture at the very end, right? So, the impact is if the yield has improved, let’s say buying point-two-five, right? Even 25%.

Depends on the number of vials the cost is and, you know, different product costs different money, you can basically compute that and get that. Now, we’re talking millions of dollars. This is not a few $100,000, [unintelligible 00:34:46] such to do that. And depends which product this model is working on. Some products are more expensive than other ones.

And sometimes rejection is high, right, so rejection is high then sometime it could be to, you know, 5%, right? If you can bring down by—cut down to 3%, like, all of a sudden you see that. So, not all the machines are kind of same rejection rate. There are different product, different—and depends on how much we are capturing, but this is a real value. This is a real value that’s being basically delivered. And we’re talking, you know, tens of millions of dollars, you know, savings over a period of time.

Brian: Got it. So, just so I can recap and imagine, like, we’ve got a factory floor with syringes and vials going through it and there’s a pile off to the right side, which is ‘suggested rejects’ or something like this. You built a data product that can help tell, should these really be rejected or are they just fine? And if you can reduce the number of false rejects the rejects that actually are just fine and get those back into production, we’re not wasting good product. And effectively, is that correct? Is that what your tool helps the manufacturing person do?

Vijay: Yeah. So, just to let you know that we are not allowed at the pharmaceutical process, once your system rejected those units, you cannot do the manual inspects and put it back. That is not allowed because that’s the GMP process, good manufacturing process. What this system does, it looks at through all the rejected material and images coming out, goes through the model, and gives you the analysis of what exactly happened. What kind of defect the system rejected.

Brian: Oh, I see.

Vijay: Now, you can take the information for the next batch.

Brian: I see.

Vijay: [crosstalk 00:36:26].

Brian: Got it. So, you’re giving insights on what do I need to change about my production process to not have that—

Vijay: That’s right.

Brian: —false—okay. And then you can measure before and after, like—

Vijay: That’s right.

Brian: On average, we have this fewer rejects now coming out, and there’s a multi-million dollar business value coming from that. I see. Great, that’s awesome. Vijay, tell me a little bit, you know, if this all sounds really nice, and all of that, okay, I’m sold, I work at a, you know, pharma company, or I work at a similar organization. I want to dip my toe in that pond because that sounds really good over there.

What do I need to do if I’m a you at another company? I’m a director of data science as well. I’ve got a team. We’re shipping stuff all the time. A lot of it’s not really getting used. Don’t really know the business impact. That way sounds neat. How do I get started? Like, what do I do? Like, literally this week or next quarter, if I’m planning for this change? Tell me how to get started? What would I do?

Vijay: Yeah. So, a couple of things that are really to be key in order to make that happen. Of course, you know, having the business strategy, what the business wants to do, what business challenges are there, right? So, as a leader, I think you want to be able to identify those challenges, where if we deliver that, then we are going to have that impact the business. Once basically that is done, think about in order to make all the models, right, you need a data, right?

And I can tell you that data has been so many challenge. So today, for example we gave earlier, not all the machines are saving the images of the rejected product that we talked about. So, before we go and deploy the solution, one of the thing that you have to do is you have to go and plumb it, the pipeline to capture those images for those rejected product on the floor. Otherwise, even deploying the model, that you don’t have anything to feed to, right? So, now we got a data challenge that we have to basically solve it, right?

So, you got to have a team of people who are really good at data engineering part, right? Because that’s really, really key. So, that’s the one work stream, we need to make that happen, right? Then the whole data science team that has to happen the model piece, right? So, computer vision is—technology in this particular case, right, so you need a expert on that.

And I think having a good team, Brian, is that what makes. People are the one who make things happen, right? You have all technology, everything else is looks good, you have a data, but the people are the one what’s going to make things happen. So, having a team with the right skill set, and follow a standard approach. We talked about the data product management concept, the design thinking really critical piece. You better bring somebody from that knowledge to do that.

You want to manage the change, as a leader. You have to be good at that. Managing the business sponsor, this will be part of the—business leaders will be part of that, right, so you want to making sure that you are a data science leader who can basically manage the change from the people that process technology and the process as well, right? So, we talked about the data, first our business value, then talk about the data, then bringing the right people into the mix, managing the change, right, to do that. I think having those elements and take a more, like, agile approach, sometimes.

You may not be able to deliver everything in one go right, so a there’s whole product roadmap that you want to develop. So, you show the success in a small piece because there’s a whole continuous investment that has to be made by [unintelligible 00:40:02], so you cannot ask $10 million you want go. You want to have a product, deliver something, now you make the case. You make the next release, here’s the next investment we want to make. You make your case to be successful that way as well, to do that. So, I think those are some of the elements as a data science leader, I think you want to be looking into.

Brian: Yeah, yea. Is there—just kind of in closing here—and that’s a great recipe for people, I think, to go try out, potentially, in their organization, if they want to, you know, dip their toe in this data product pond, so to speak, is there’s one thing about this kind of topic, this whole conversation we’ve been talking about something that you learned that you wish you had known a lot earlier? Something that you had to maybe learn the slow and hard way that you might share with people who are kind of on a similar journey, but maybe they’re not quite where you’re at yet with what you’re doing?

Vijay: Yeah. I mean, there have been many, many things. One thing I can tell you is that at one time, was prior to Merck, but I learned my lessons, actually, where I developed a solution and I thought the solution is going to be, you know, easy to deploy. And I never considered as the leader of the team that they’re going to be challenging it when we go to deploy it, right? And I’ll give you a specific example where whereby we’re trying to deploy this solution, and the data that was feeding was for the remote place, the very remote areas, more like an jungle, right?

And the speed the data was supposed to be coming to this model that was being posted in the cloud, there was a lag between the data because the communication line was really weak, right? So, I couldn’t implement the solution because the latency in the data just could not basically make on the time that we wanted to that. So, all the work and hard work that went, that’s a really bad situation to be in. So, I would say that when you develop the solution, you want to consider can this be deployed? Can you face—and I will specifically look for the data side of it. So, you can have a model, but now once you deploy the model, you got to feed the data continuous.

Can you see any challenges in that aspect of that? I think that comes to my mind in a very technology point of view. And other element. I think as I mature, you know, the data science leader, I think the people aspect, I pay much more attention to that. I mean, how do you make people motivate to be part of your team? What exactly are they giving you in the bag, right?

So, looking after their interests, what their aspirations are, and you want to make sure that you got a strong team and motivated team to basically deliver. And human spirit is something, you cannot believe how stretchable it is. If the people are motivated, you have less resources and less technology, they will still achieve it. And in the new, you know, time, I think I pay really a lot closer attention on those elements. In the past, I wish I had done more of those.

Brian: Yeah. I hear this a lot. It comes up all the time. And it makes sense, right? We’re all humans, we’re all feelings and I think sometimes the technology and the data and all these all the possibilities of what we can do now especially, you know, in this kind of modern era that we’re in where we don’t have some of the technical constraints, it sounds like, “Why wouldn’t somebody want this?” Like, “Why would someone ignore this? Like, I can’t imagine that world.”

And yet, it continues to be a challenge all the time for many organizations. So, I think it’s great insight to really be focused on that. I mean, it’s a big part of my work, too, is how much the human behavior part and how do I incentivize someone to change? And as I recommend to, you know, my seminar training participants and stuff like that, part of this and what’s known to work well is, how do you make this person’s life better within the scope of how they already live and work and do their stuff? Like, don’t try to introduce a change; try to introduce an improvement to the way they’re already doing it now so that the tax is less, right?

You’re trying to minimize any type of imposition on them, and make it so much obvious why this is better without that disruption. That’s really the key, I think, to the adoption piece there and really centering it on how is it going to be better for them? Because if they don’t feel how it’s better for them, then it’s just another hoop to jump through, right? Another form I have to fill out when I want to get my expense reimbursement. Another, like, oh, enrollment for whatever, like, it’s yet another one of those things, another tool, another task, another report.

And the more we can reduce those taxes on people, the more they’re going to say, “Wow, like, they really build great stuff for me and I feel more empowered.” Like, it does come down to that human thing, I think.

Vijay: Yeah. Well, said. Absolutely.

Brian: Vijay, it’s been great. Where can people get in touch with you? You hang out on LinkedIn, social media, anywhere? If someone wants to kind of follow your work and postings or anything like that, tell us where.

Vijay: So, I’m on LinkedIn. I think folks can find. You know, I always mentor and help people succeed in their career. And so, I mentor people inside my work and outside, I help them out. So, I can be easily accessible on LinkedIn if someone wants to brainstorm an idea, a data analysis strategy, or anything else, you know? And sometimes I see the interactions, and not that I’m giving something to them; I’m learning from them as well. So, I think it’s a win-win. If anybody wants to brainstorm, you know, new ideas, I think that’s the best way to learn for both sides.

Brian: Cool, cool. Well, we’ll dig out your LinkedIn link and we’ll put that into the [show notes 00:45:32] and everything. So, thank you again for coming on Experiencing Data and sharing your thoughts and ideas. Appreciate it.

Vijay: Thank you, Brian.