Today, I’m chatting with Adam Berke, the Chief Product Officer at The Predictive Index. For 70 years, The Predictive Index has helped customers hire the right employees, and after the merger with Charma, their products now nurture the employee/manager relationship. This is something right up Adam’s alley, as he previously helped co-found the employee and workflow performance management software company Charma before both aforementioned organizations merged back in 2023.

You’ll hear Adam talk about the first-time challenges (and successes) that come with integrating two products and two product teams, and why squashing out any ambiguity with overindexing (i.e. coming prepared with new org charts ASAP) is essential during the process.

Integrating behavioral science into the world of data is what has allowed The Predictive Index to thrive since the 1950s. While this is the company’s main selling point, Adam explains how the science-forward approach can still create some disagreements–and learning opportunities–with The Predictive Index’s legacy customers.

Highlights/Skip to:

• What is The Predictive Index and how does the product team conduct their work (1:24)
• Why Charma merged with The Predictive Index (5:11)
• The challenges Adam has faced as a CPO since the Charma/Predictive Index merger (9:21)
• How Predictive Index has utilized behavioral science to remove the guesswork of hiring (14:22)
• The makeup of the product team that designs and delivers The Predictive Index's products (20:24)
• Navigating the clashes between changing science and Predictive Index's legacy customers (22:37)
• How The Predictive Index analyzes the quality of their products with multiple user data metrics (27:21)
• What Adam would do differently if had to redo the merger (37:52)
• Where you can find more from Adam and The Predictive Index (41:22)

Quotes from Today’s Episode

• “Acquisitions are complicated. Outside of a few select companies, there are very few that have mergers and acquisitions as a repeatable discipline. More often than not, neither [company in the merger] has an established playbook for how to do this. You’re [acquiring a company] because of its product, team, or maybe even one feature. You have different theories on how the integration might look, but experiencing it firsthand is a whole different thing.  My initial role didn’t exist in [The Predictive Index] before. The rest of the whole PI organization knows how to get their work done before this, and now there’s this new executive. There’s just tons of [questions and confusion] if you don’t go in assuming good faith and be willing to work through the bumps. It’s going to get messy.” - Adam Berke (9:41)
• “We integrated the teams and relaunched the product. Charma became [a part of the product called] PI Perform, and right away there was re-skinning, redesign, and some back-end architecture that needed to happen to make it its own module. From a product perspective, we’re trying to deliver [Charma’s] unique value prop. That’s when we can start [figuring out how to] infuse PI’s behavioral science into these workflows. We have this foundation. We got the thing organized. We got the teams organized. We were 12 people when we were acquired… and here we are a year later. 150+ new customers have been added to PI Perform because it’s accelerating now that we’re figuring out the product.” - Adam Berke (12:18)
• “Our product team has the roles that you would expect: a PM, researcher, ux design, and then one atypical role–a PhD behavioral scientist. [Our product already had] suggested topics and templates [for manager/IC one-on-one meetings], but now we want to make those templates and suggested topics more dynamic. There might be different questions to draw out a better discussion, and our behavioral scientists help us determine [those questions]... [Our behavioral scientists] look at the science, other research, and calibrate [the one-on-one questions] before we implement them into the product.” - Adam Berke (21:04)
• “We’ve adapted the technology and science over time as they move forward. We want to update the product with the most recent science, but there are customers who have used this product in a certain way for decades in some cases. Our desire is to follow the science… but you can’t necessarily stop people from using the stuff in a way that they used it 20 years ago. We sometimes end up with disagreements [with customers over product changes based on scientific findings], and those are tricky conversations.  But even in that debate… it comes down to all the best practices you would follow in product development in general–listening to your customers, asking that additional ‘why’ question, and trying to get to root causes.” - Adam Berke (23:36)
• “We’re doing an upgrade to our platform right now trying to figure out how to manage user permissions in the new version of the product. The way that we did it in the old version had a lot of problems associated… and we put out a survey. “Hey, do you use this to do X?’ We got hundreds of responses and found that half of them were not using it for the reason that we thought they were. At first, we thought thousands of people were going to have deep, deep sensitivities to tweaks in how this works, and now we realize that it might be half that, at best. A simple one-question survey asked about the right problem in the right way can help to avoid a lot of unnecessary thrashing on a product problem that might not have even existed in the first place.” - Adam Berke (35:22)

Links

• The Predictive Index
• Adam on LinkedIn

Transcript

Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today, I have Adam Berke on the line, a Chief Product Officer at The Predictive Index. Adam, how are you?

Adam: I’m doing very well. Thank you so much.

Brian: Yeah, this was a funny episode because I think it was Eric Huber, right, put us in touch. So, I was playing a gig with New Bedford Symphony, I think, and my friend down there is a percussionist as well—we’re both percussionists—and somehow we got to talking about what he was up to, besides playing music, and he mentioned he was working at Predictive Index. And I had heard of you all before. And so anyhow, we’re now here to talk about product at Predictive Index, through the percussion world. So anyhow, I thought that was kind of funny. So, thank you to Eric for the introduction. Tell us a little bit about what is the Predictive Index first of all, and then we can kind of jump into some product nerdery.

Adam: Great. Yeah, I thought you were going to stump me with a music theory question—

Brian: No.

Adam: So, better to start off with the Predictive Index. So yeah, the Predictive Index is actually a 70-year-old company that helps people hire and manage people based on behavioral science. And so, at the core of the product, we consider ourselves a talent optimization platform. And so, that involves kind of the range of employee experiences from, yeah again, finding the right person for a specific job, making sure that they fit in well with the team that they’re joining, giving people vocabulary and understanding of the dynamics that might emerge on that team, and then helping to guide the employee-manager relationship, again, instead of by, like, dark arts and guesswork, through behavioral science. And at the core of the Predictive Index is a behavioral assessment which helps people understand how their coworkers and how they themselves, might relate across different dimensions.

And there’s four of these dimensions in the Predictive Index. It’s a very quick assessment, which is one of the key differentiators. And again, I mentioned the durability of the company. This kind of assessment and methodology has been battle tested significantly over the years. So, you can think of it as sort of like a behavioral assessment that you might be familiar with, but highly tailored for this workplace use case.

Brian: Got it. Got it. Two things came to mind immediately. One was, I was thinking about, I think it was Daniel Kahneman’s work—I could be wrong about this—in the Israeli military, about predicting who would make good officers, and I’m curious if any of this is rooted in that. And then secondly, I was thinking about, I wonder if they’re using—are they training models on this to predict stuff, and if they are, then where is the data coming from? And is it coming from past employment history, which is probably somewhat biased towards—especially in, like, corporate America—white men. So, my brain [laugh] started rolling into all that. So, I’ll just throw that at you and let you talk a little bit. I’m not trying to pick on your product, but I’m just curious.

Adam: Yeah, we’re pretty outside of that debate. You know, I mentioned the 70-year-old history. So, it actually ended—I’m not actually familiar with the Kahneman work on that specifically, but yeah, no, the founding story mimics that pretty closely. It was a returning vet named Arnold Daniels from World War II who had similar questions around, you know, leadership in the military, and specifically his experience with leadership in the military, and why did his, you know, platoon perform the way it did. And so, you know, he carried over that work into the academic setting, and did research on it, and developed the methodology then. And so, that’s why it goes back the 70 years.

And that’s very much prior to large language models, so you know, it’s not based on that type of data set. And as I mentioned, you know, over the years, the science has evolved as we learn more about it. And today, it’s really, you know about—and you know it is kind of—a lot of these methodologies have sort of converged. You know, I mentioned that there’s four primary factors, you know, that might sound familiar from other methodologies you’re familiar with. So, yeah, and, you know, course-correcting it over the years for, you know, increasingly diverse workforce is certainly something that’s always top-of-mind and something that we do, and think about, especially as you’re thinking about how to apply it into people management, which is the part of the product that I come from.

Brian: So yeah, and one of the things I wanted to talk about, there was a merger, correct, with Charma.

Adam: Yeah, that’s how I came into the business. So, I co-founded a company called Charma, which was around guiding workplace relationships and really building the workflows around the employee and manager relationship. So, guiding one-on-ones and helping provide people with a structure for having a productive one-on-one, how to give feedback, helping to guide giving feedback, helping to take friction out of some of these really high value but often, you know, rejected, or postponed, or procrastinated things that every manager knows, or the types of things that they should be doing, but because there’s this added mental overhead, or added, yeah, mental friction to do these things, they don’t happen as frequently as we’d like.

And so, Charma was meant to be kind of putting those best practices on guide rails. What we were missing, though, you know—and this is something that I’m sure we’ll talk a lot about—from a product perspective, was, you know, where to land on the spectrum of being highly opinionated versus flexible. And so, you know, in these workflows that we were helping with, setting goals, giving feedback, recognition, having a one-on-one, having a group meeting. Do you want to be highly prescriptive in that or, you know, you could go to the far other end of the spectrum. And oftentimes, what we were replacing was a Microsoft Word doc, or, you know, Google Doc for the one-on-one.

And so, that’s obviously highly flexible. You can put anything you want on that page. And you know, you can use any process you like for having a meeting or processing that meeting agenda, tracking action items. And so, you know, we were trying to move the spectrum a little bit towards something that was more prescriptive, that was more guiding people, instead of just a blank page, things like templates, and suggested topics, and things of that nature. But what the Predictive Index brought to the table that was really appealing to us was this behavioral science. And that was just, like, entirely differentiated in the space that we were playing in.

So, having not just, “Okay, well, here are some topics you might want to talk about,” getting down to, “Here are some topics you might want to talk about, given the type of person that you’re talking to, and the type of person you are, and therefore how you might relate to one another.” It’s just a whole additional layer of precision that you could get to. And so, we were, you know, a fairly small startup growing rather quickly. Predictive Index, as I mentioned, had this amazing legacy and proven science and obviously, distribution, which is often the appeal of, you know, that’s kind of the quintessential match for an acquisition. You know, we were pushing technology quickly, and they had the distribution that we, you know, wanted to bring the product through.

And those things that we were building, and kind of that post-hire employee experience were kind of like all the things on Predictive Index’s roadmap, and so we were able to very quickly accelerate that, and then plug in their behavioral science to these workflows. And that was really the first year of work. And, yeah, I joined as, initially, the kind of overseeing that business unit. So, the company that was acquired was going to be one module within the Predictive Index space. And, you know, after a few months, I ended up overseeing, now, the whole Predictive Index product as the chief product officer.

Brian: And I think this, like, merger, is something I wanted to talk about. Just how do you bring together product teams, product processes, obviously, there’s always the staffing and making all that stuff work. And just high level to connect the dots for listeners. I’m guessing here that the business strategy or thinking here was more or less, Predictive Index sounds like it sits in the early part of the life cycle, which is during the hiring phase, but what’s the value it can add after someone has been hired? Charma kind of kicks off at that point where you’re now an employee and you have a manager.

Okay, so that was kind of the high level the business thinking there. So, you can kind of take this where you want, but what kinds of experiences have you had so far, either in the—I’m curious, like, challenges and learnings from the merging of the product, like, making the product a single product, or the team side, the actual processes and product manage-y stuff and all of that. What have you learned so far going through all this?

Adam: Yeah, acquisitions are complicated, and so I’ve been a part of, you know, basically launching three different companies—this is my third—and I hadn’t gone through an acquisition like this before. And I think that’s typical, right? And outside of a few select companies, you know, a Cisco, or maybe a Microsoft, or like, you know, you could probably list them on one hand, there are very few companies that have M&A as, like, a repeatable discipline. And so, for the vast majority of acquisitions, you know, including ours, it’s most likely the first time that the entrepreneur or the founder, and certainly the startup, has ever been acquired, and it’s often the first one or two times that the acquirer has done an acquisition. So, neither side has, like, an established playbook for how to do this, ignoring the fact that, I think even in those larger companies, each acquisition is somewhat their own snowflake and has their own considerations.

And you know, you’re bringing it in because of the product, or you’re bringing it in because of the team, or maybe even one feature. You have different theories on how the integration might look. So, you know, just starting with that context, which I think I knew intellectually, but experiencing it firsthand is, like, a whole different thing. Which makes me realize that, you know, maybe the most important thing is not necessarily, like, knowing all of the, like, best practices, but entering into the process with, like, good faith, and a good relationship, and trust because you’re going to have to, like, adapt on the fly, and there’s going to be a ton of different things that just, like, are unclear, or fall between the cracks, or require a new way, a new process.

I mean, just thinking about, like, my initial role as an example. So, my initial role didn’t exist [laugh] in the company before. All of a sudden, it’s kind of like plopped in and the rest of the whole PI organization knows how to get their work done before this, and now there’s this kind of new executive. And when do we talk to him for stuff? What’s he even doing? And so, you know, there’s just tons of stuff like that if you don’t go in with, like, you know, assuming good faith and, you know, trust and willing to work through the bumps, it’s going to get messy.

And it certainly did at times, but here we are a year and a couple months later, and you know, I think we had really ambitious goals for the first year that, you know, some got done, some didn’t get done, but I think, you know, I would have to say we all feel pretty good about where we ended up after a year. And now looking back after it’s like, yep, that’s pretty much what we should have expected. Like, we integrated the teams, we integrated and relaunched the product, we now have customers on the product, so the product is now a module within the PI suite. And this was helpful, you know, going back to your initial question, that the business case and the product integration had a pretty clear story from the outset.

So, prior to the acquisition, PI already had four modules: so hire, which was about how to hire; Inspire, which did have a lot of the content around how to manage people, so it looked at the behavioral assessments and how they matched up; Design, which looked at your behavioral assessments within a team, so it’s like team design; and Diagnose, which was like engagement surveys. And so, now we could add Charma, which now became Perform—PI Perform—as a distinct module. And so, pretty much right away, you know, there was some re-skinning, and some redesign, and some back-end architecture that needed to happen to just make it its own module, but then right away, it could work. But over time, from a product perspective, you know, what we’re trying to do is deliver that unique value prop that we talked about at the outset, which is not just, like, fusing our workflows in, but really where the—if I may use the jargon—synergies comes into play, is when we can start infusing PI’s behavioral science into these workflows. And so, those are the things that it’s new stuff, and it’s not something that clearly exists out in the world, so figuring out what are the places where that makes sense, what are the places where that doesn’t make sense, is something that we’re figuring out.

But I think we’re on, like, a really good track with that because we have this foundation. We got the thing organized. We got the teams organized. So much, you know, people stuff, even with a fairly small team, I think we were like 12 people when we were acquired, but figuring out how all that fits into the org chart, like I mentioned. And yeah, here we are a year later. I think 150 or so new customers have been added to PI Perform, probably more at this point because it’s accelerating now that we’re figuring out some of these dials in the packaging. So yeah.

Brian: I’m guessing, when you bring intelligence—or insights, in this case, from the behavioral science side—do you ever find it difficult to get customers to believe this information, and to act on it for whatever reasons, either just bias that they have or denial [laugh] or I’m curious how much of the behavioral insight part is a challenge to make believable within the experience, to make it feel valuable, trusted, useful, all of that aspect.

Adam: That actually isn’t really where the bottleneck is, as far as we’ve discovered. When people buy PI, they usually go all-in on—they buy it because they believe in the science. And you know, again—and PI is one of those things that people bring from, like, company to company. We see a ton of that. And so, once your brain starts orienting around the science, a lot of companies really do, you know, become true believers in it, and it really helps to give—it becomes like a vocabulary within the company.

And, you know, it’s obviously true within PI, you know you’re talking about, “Oh, this is, for all those low D people out there”—you know, low D being, you’re maybe a little less in the detail—“So, here’s some details that you might need, or you might have missed or glazed over,” or I’m trying to—so it becomes part of that vocabulary. We have kind of a visual representation of people’s behavioral profile that oftentimes people put on their doors in their office. So, you know when, like, when you go in, like, it’s a reminder, the type of person that you’re engaging with. And the people who don’t believe in it are not going to be the people who kind of retain onto the product, like, you know. So fortunately, it works, [laugh] so when people start using it, they okay, yeah, this worked. This helped take some of the guesswork out of hiring. That’s usually the first use case, and we go from there, and then, as I said, it becomes sort of a language in the organization.

So, you know, the bottleneck, I think, is less about the skepticism and more about, like, how and when are the right points of intervention to make the data useful, without it becoming, like, disruptive or adding friction to whatever they’re trying to get done. And especially as we’re thinking about these new workflows, it’s like, okay, you have a one-on-one with someone, do we want to, like, put some screen takeover that, like, you know, you have to, like, review some stuff before you can even start your one-on-one? That’s, like, an extreme example—we wouldn’t do that, but just for the sake of discussion—that would be a very deep intervention. Or, you know, do we put something like three clicks deep. Like, if you want to learn more of this person, click here. It’s like, okay, well, that’s very subtle, but you’re not going to have the impact that you want. So, figuring out where to land on that spectrum is more of the nuanced design and product stuff that we’re thinking through. If you’re a doubter that this is useful, then, you know, all that is a moot point [laugh].

Brian: Yeah, I guess I was thinking… I mean, one of the common challenges with some of the B2B analytics products, or products that are using any type of intelligence, and things like this, is the fiscal buyer sees the value in it. So, in your case, they see the science, a head of HR, or I’m not sure who would be resp—talent—who would be responsible for making a buying decision, but then you have ICs, or mid-level managers that are using it day-to-day on the hiring, and maybe their incentives are misaligned with, like, get the best people. It’s like, no, I just need to fill the jobs as quickly as possible. Whether they’re good or not, that’s kind of someone else’s problem. That’ll be figured out later.

These kind—I’m just making up an example. I’m not saying that’s true in your case, but I’m always kind of curious about the way the product is positioned for the buyer versus the end-users, and is there a disconnect sometimes between, you know, what’s right for the company versus what’s right for the individual user? And [laugh]—

Adam: Yeah well, you bring up a good point. I mean, and like, what is the driving motivation for that person that is saying, “Okay, well, you want to find the right person. I need to put a butt in a seat.” It’s because there’s a latent assumption there that, like, finding the quote-unquote, “Best,” or whatever is going to take more time. And I think that’s one of the areas where PI has really nailed the sort of like Pareto optimal assessment.

There’s definitely, like, other assessments out there that are, like, much deeper, but you literally need a PhD to administer it to make sure it’s—and the fact that our assessment takes about six minutes, and the information that you get from that six minutes is a lot of signal kind of takes some of those concerns out of the equation. And you know, when you’re a hiring manager, it actually solves that latent concern that you mentioned that, like, oh God, this is going to be such a ordeal, and I needed someone yesterday. It’s like, well, wouldn’t you like to see before you even schedule an interview with someone, how the various candidates map to the type of job that you’re hiring for, and how you kind of described the requirements of the job? So, you know, someone in customer support is going to probably, you know, and in the past, has—successful people have had a different level of extroversion, let’s sat, than, you know, an engineer where you know a high level of extroversion might not be as valuable, depending on the makeup of the rest of the team, right? If the rest of the team is not extroverted, then it might be useful to bring someone onto your team that’s extroverted. But again, all of this from the point of view of the hiring manager, can just be seen as they think about who to talk to. So, it’s just another really valuable, succinct data point that they can look at instead of trying to read the tea leaves of a resume. You know, brings its own biases to the table.

Brian: Sure, sure.

Brian: Let’s jump in a little bit to the—I’m curious. You were talking about the actual design of the experiences here, bringing in some of the scientific insight that you have around—it sounds like you’re trying to maybe nudge people or encourage certain types of behaviors at the right time and all of that. Who gets involved? Like, do you have, like, a recipe for a core team when we do this, we have a behavioral scientist, and we have a UX person, and a product manager, and a technologist? Like, do you have, like, a playbook for how you go about doing that, and what roles are necessary to bring those insights into the product?

Adam: Uh, yeah. I mean, you kind of listed them.

Brian: [laugh].

Adam: Yeah, so we’re set up—yeah, like you know, the product team has the roles that you would expect with a PM, researcher, UX design, and then, yeah, the one additional role that might be somewhat atypical is a PhD behavioral scientist on staff to make sure that, like, that’s not a validated use of the science or, especially now, there are situations where, you know, we don’t have the content, the science-based content, for the use case that we have in mind. So, a very simple example would be those one-on-one discussion topics. That was kind of our bread and butter, helping people have better one-on-ones. And we had suggested topics and templates prior to the acquisition, but now we want to make those templates and suggested topics dynamic, based on the what we, as we call it, reference profile of the two participants in the one-on-one. So, you know, if I’m your direct report and I’m highly introverted and you’re highly extroverted, just as a very, like, just superficial example, there might be different questions you’re going to have to ask me to draw out a better discussion.

And what those questions are, our behavioral scientists would have to help us determine. And there’s a lot of those, right? We didn’t have those suggested topics, like, laying around. You know, that person has to look at the science, look at other research perhaps, calibrate, and those are things that we can also sanity test, and look at from different angles before we implement them into the product.

Brian: Got it. Got it. Are there ever rubs or friction between the science says that we need to do it this way to make it valid, but your product sense, or maybe your user experience designer, who’s, like, “I live and breathe with HR managers all day long. If we put this nudge here, this question here, we’re pretty sure that they’re going to drop. That’s going to just end this. They’re not going to give this a chance because the ask is too big.”

Or I’m kind of making up something, but I’m always curious when the facts of the world, as known by the SME, who’s there, who knows the data the best, sometimes conflicts with the reality of what would be accepted, what’s actually going to be viable versus what’s scientifically correct. Those are not always the same things. I’m just curious if those rubs come up, and how you navigate those.

Adam: Yeah, there’s a bit of that, but I’d say the bigger rub that comes to mind in a 70-year-old business—and I mentioned how, you know, we’ve adapted the techno—the science over time, as science does, it moves forward and we learn more, and we are able to validate various things and invalidate various things. And so, we want to update the product with the most recent science, but there are customers who have used this product in a certain way for, in some cases, decades. We also didn’t talk about it, but we have a Certified Partner Network. There’s about 400 Certified Partners who are strategic partners with our clients and help to implement Predictive Index into those organizations, and provide consulting and training on how the methodology works, and how to implement the software, and where to use it. Those folks have been oftentimes using the product and the science in a certain way for decades.

So, the friction is, I think, less internal, where it’s like, oh, this is the best way to—you know, we have every—you know, our desire is to follow the science. It’s a pretty clear guiding—that’s pretty clearly the correct guiding principle to have, but you can’t necessarily stop people from using the stuff in a way that they used it 20 years ago. And you can’t necessarily, you know, enfor—you know, if you have a partner network, you can’t necessarily enforce how they talk about something, or how they use it. And so, those are the things that are a little bit more out of our control, and where we sometimes end up with disagreements, where, “Hey, you changed this thing in the product,” or, “I used to be able to see this data point, and it’s no longer there.” And for us, it’s like, oh, well, we found that it doesn’t actually measure the thing that we thought it did. And, you know, there’s a whole discussion around that, you know? It’s like, “Okay, well, where is it? I mean, I’ve been using”—well, you know, like, okay, what do we do there? And so, those are the trickier conversations, I’d say.

Brian: Yeah I can imagine that being really hairy, especially when you’re pulling stuff out, that maybe a business has been—like, this is our master KPI that goes on the CEO’s dashboard, and you’re like, yeah, it doesn’t do what we thought it did [laugh] because there’s new science, you know? Like [laugh]—

Adam: Yeah, I mean, there’s a very specific, one of those that comes to mind for me right now. But even in that debate, it’s interesting because the partners, for example—or the clients—when we say, “Hey, that’s not scientifically valid,” sometimes they indicate, “Oh”—and there’s just one example that comes to mind for me—“Well, yes, it’s not scientifically valid for the thing that it was used for, but we used it in this other way, I.e., to make sure someone was actually spending time on the,” you know, think of it as, like, the number of responses someone selects in one of our assessments as a metric.

What does that metric measure? Well, there might not be a whole lot of science behind certain things, but it could measure—even you don’t need science to know, like, oh well, if they only responded to two things, that’s something useful to know. They weren’t really engaged in it. And that might not be a deal breaker, one way or another, but it’s useful to know that this person wasn’t really engaged in the assessment, you know?

Brian: Right.

Adam: So that’s, like, one of those things, like, okay, oh, we didn’t think about that. We might actually want to surface that metric in a different way so that it’s not construed as a, you know, as a scientific data point that measures X, but you still know what it is because it gives you this additional insight. And so, that just comes down to all the best practices you would follow in a lot of features and product development in general, in terms of, like, listening to your customers, and asking that additional why question, and trying to get to root causes.

Brian: Yeah, yeah. This kind of dovetails nicely into my question for you on how you measure user experience quality versus the products quality in terms of fiscal, right, like, you know, usage numbers or seat licenses used, or payment, you know, like, [laugh] is the revenue coming in the door versus the user experience of it. How do you know whether or not you’re delivering a good experience in the product? Do you have measurements for that?

Adam: Oh, of course. It varies between the pre-hire experience and the post-hire experience, obviously. You know, in the pre-hire experience, we have a whole range of things that we use. And we have over 10,000 clients, so you know, we have a good… we have a pretty solid data set there. There’s all sorts of things that we can use to determine. And a lot of it, though, boils down to, if they’re hiring, you know, how many jobs are they creating, and then how many assessments are they sending out, right? Like, how many people are they assessing?

And those two things matter because, like, if you’re not doing a lot of hiring, you’re not going to be sending out a lot of assessments, and you know, the ratio of those things needs to be weighed—my kind of general point of view on this is that people tend to, like, overcomplicate these formulas, and you know, just provide—at the end of the day, you end up with some metric of engagement, you know, 3.875 because they did this little thing, and you multiply the number of that little thing they do times this other thing. And, you know, you end up with these very complex metrics that’s, like, hard to figure out where the bottlenecks are. So, you know, for us, it’s like, did you create a job target—so basically, like, is there a job you’re trying to fill—and how many assessments did you send out?

And then post-hire, you know, in the Charma suite, it’s sort of similar. Like, A, are you inviting people into the product? If you’re inviting people into the product, B, are they using it? And we have a definition for that, you know? You have to put content into the product, you know? A discussion topic, a comment, a piece of feedback, a kudos, recognition, anything that’s, like, putting into the product. And then did you use it again? And that’s just—you know, within X days.

Because we know that there’s a new habit or behavior that needs to emerge, and so, you know, if you do those three things, great, and if you don’t do one of those things, we kind of know exactly where the bottleneck was. And it doesn’t really need to be that much more, I don’t think, complicated than that. You could probably get it more precise, but you’re getting diminishing returns, and in those diminishing returns, you’re running the risk of, like, confusing the thing that you’re measuring, you know, and you get in these weird situations where I can have someone do a bunch of, like, low-value actions that add up to a good score, but they didn’t do the key thing that we really need them to do. And so, I’m a big believer in kind of boiling those things down.

Brian: Sure, sure. So, it sounds like you’re using analytics, right, trackable things within the product as a proxy for what the experiences that users are having. So, I guess what I was curious more about is, like, whether you had five open recs or you had 50 this quarter as an H—I’m guessing a HR hiring manager is a common user. Is that fair?

Adam: Yeah, yeah. On that side of the product. Let’s stay there. Yeah, recruiter [unintelligible 00:30:36].

Brian: Yeah, on the hiring side. So, do you do any measurement to know how is the experience of figuring out what screener assessment to use to post the right kind of job? I don’t know if you guys get involved with guiding the actual requisition, how it’s worded, and all of that, the receiving of all the inbound resumes, or whatever the hiring process is like, the actual experience of, like, was that good, acceptable, delightful, exceptional, these qualitative things versus completion rate? Do you do any measurement on that, on terms of the actual benefits to the person, the HR manager, so that you know, they did 80% of them, but were they, like, hating every minute of it, or was it like, this is so easy, it’s just mindless. It’s just, like, I don’t even think about it because it’s so great. Do you do any measurement on that side at all?

Adam: Yeah, of course. You know, we do all the things that I think you’d expect there, you know, in terms of user research, you know, especially before we build a new feature, and getting feedback on that getting, you know, obviously, the idea to even build that feature in the first place likely came from either customers or partners, and bubbled up often that way. I think we’re often more bottoms-up in that respect. We try to be, certainly, I do. And then, yeah, you know when stuff’s out in the wild, getting user research, watching usage sessions, and that is the art and the science of product development, right, like balancing that, you know, that subjective user experience, user feedback side of things with, you know, what do they say versus what do they do, and disentangling those things.

You know, I’d say, often the data points us where to look, right, and says, “Okay well, hey, this company sent out this many assessments for the first role they hired for. The second role they hired for, you know, a rec showed up and they didn’t send out any assessments.” That’s probably… okay, something about that first experience didn’t go well. And so that, you know, can point us in the direction of, like, okay, well, what didn’t go well? And start asking questions from there.

Brian: Who’s involved with those sessions? Is it just the user experience team or the research team? Or do you, like, pair up the behavioral scientist with them, or the product—like, what’s that look like? Paint a picture just with those sessions.

Adam: There’s a whole variety of ways that could play out depending on, you know, what we’re doing, if we’re trying to diagnose a problem versus we’re trying to build a new feature. You know, again, building the new feature is often where science would be involved proactively. Those types of things will get to them if there’s a problem that’s identified, and then there’s, like, a solution that, oh, okay, we have an idea for a solution. Science might need to—you know, oftentimes we develop a pretty good, you know, just internal barometer, like, okay, this is not, like—this doesn’t really affect—I mean, a lot of the things that we might need to fix have nothing to do with the science, just like any other product that’s out there, you know? The onboarding flow is confusing, or the screen, or there’s a modal that blocks this button, you know, all the things that you end up solving.

So, you know, there’s loads of stuff that doesn’t even touch science and would just be handled in the traditional product development, user research-type session where we’re, you know, trying to do a certain number of interviews. We’re starting with low-fidelity stuff, working to high-fidelity stuff, following all the kind of typical playbooks that pretty much anybody could read a blog post about [laugh].

Brian: I think for some… some of the audience at this show that are not necessarily working in technology companies, some of these things are not table stakes—

Adam: Yeah.

Brian: —that both the roles involved, the work that’s being done, the roles of the people doing it, and that kind of thing. So, I think it’s good that you talk about this, even if it’s bread and butter, you know, for you all.

Adam: Yeah. So, in that spirit, there are—so, yes, there are the, like, typical things that we would control that are, like, the table stakes, you know, the bread and butter, the traditional or the, you know, most expected way. And then, so given that those are somewhat table stakes, I think there are, like, novel ways that you can get input, and that we do get input. And you know, some of those would include technologies, like, you know, at Charma, we used a product called Fullstory, and at PI, we used something called LogRocket, you know, which basically gives you screen recordings of people’s usage, and kind of identify frustrations, or drop offs, or things like that. Those can be canaries in the coal mine, often where you can identify problems or errors in the flow.

We do—maybe to a fault—a bit too—but it’s a useful data point—various customer and partner surveys. And so, you know, I can think of an example right now where because of the response rate of one of those subjective—so we had a problem where we know peop—we’re doing an upgrade to our platform right now, and we’re trying to figure out, just to make this really concrete, how to manage, user permissions in the new version of the product. The way that we did it in the old version had a lot of problems associated. It was very manual, it was very organic, you were literally playing around with, like, this folder tree structure to figure out who had visibility to what, and that’s not a very common paradigm out in the market.

And so, we just looked to see—and a lot of people didn’t use that at all. They were just kind of fine with defaults, but there was a big chunk in the middle that we just didn’t know how they were using this aspect of the product. And we just assumed that anyone that had done anything with these manual folder tree structure were people who used that feature. But we were confused; it was thousands of customers. Like, what are they doing? And it was very hard to, like, look at their setup and say, “Oh, they’re clearly trying to do X,” or, “They’re trying to do Y.”

And so, that was a case where, yeah, we put a survey, like, “Hey, do you use this to do X?” And got hundreds of responses, of thousands because it was a pretty quick and easy survey, and found that half of them were not using it for the reason that we thought they were. And that has totally reset our priors on how we might migrate those people to a new experience. At first, we thought thousands of people were going to have deep, deep sensitivities to tweaks in how this works, and now we realize that it might be half that, at best. So, another example of a simple one-question survey asked about the right problem in the right way can help to avoid a lot of unnecessary thrashing on a product problem that turns out might not have not even exist in the first place.

Brian: Prompting in the survey. How you ask the questions is a, [laugh] that’s a whole science as well, a whole social science is behind that [laugh].

Adam: For sure. I mean, but you know, even you know, if you get that 80% right or 75% right, you know, you can go further. You can take it further and say, “Okay well, hey, that is not what we expected. Let’s dig further.” And you know, that’s what we’re doing. We’re not just taking, “Oh okay, well half a [unintelligible 00:37:39], you know, doesn’t matter.” You know? But it’s like, wow. Okay, that’s not what we thought, and there’s pretty big implications to that. Now, okay, we have this population. Let’s see if we can talk to a couple of these people and go further. So, yeah.

Brian: Cool, cool. Any closing thoughts about this, like, merging process? I don’t know if you reflect back 14 months or how long it’s been. If you were writing yourself a guidebook for, like, how to handle the first year, would there be any things that you would put in there now, like, explicitly? Like, all right, don’t do that again. Or anything you want to share, either on the merging, like, bringing the products together, or on the process, people side? I’m more curious about the product, but kind of open-ended.

Adam: It’s really interesting, and obviously something I’ve thought a lot about. I think it’s going to be hard to say, you know, on a very granular level, do do this or don’t do that because I do think each—and I’ve been trying to think of those things, obviously, because I want to be able to, like, remind myself, or give advice, or whatever the case may be. And so, much of it is dependent on, yeah, the specifics of the cultures, of the rationale, so I don’t want to evade because—but I what I’m—all I’m doing is just saying to come back to the highest level point of, like, you got to go into it with trust. And I mean, the maybe one specific actionable thing would be to communicate that really clearly, broadly in the organization, that, hey, there’s going to be bumps and there’s going to be ambiguity, and to be prepared for that, and really message that out. Because there can be an assumption that, like, oh, up the org chart, they’ve done all the work on this. They know exactly how this is going to go. They know who’s going to slot where, how it’s going—you know? And being humble in that, hey, we’re going to have to figure this out together, I think, is probably a good message if you’re at a company that doesn’t do dozens of these.

And then secondarily, I would overindex on the fact that people tend to not like ambiguity, and to try to resolve ambiguity quickly. You know, I’ll give an example that. You know, we kind of said we took, like, a 50/50 approach in our acquisition, where we had a pretty clear vision for how the product was going to slot in. This was going to be, you know, integrated as quickly as possible into the core product. This wasn’t going to be, like, Instagram is to Facebook. It wasn’t going to sit outside in its own brand, it was going to be more integrated.

But we kind of said, hey, you know, we’ll figure out the exact reporting structure and the people roles later, and change titles later. And that one… [clicks tongue][sigh] I could be—if we did that faster, I’m sure I could be coming here second-guessing that as well, and being like, “Oh, we slotted all these people into the wrong places, and you know, should have just waited, just let the dust settle.” And so, I’m sure, you know, there’s a counter factual to this that’s wrong also, but you know, I think there was a lesson there where just that ambiguity and introducing these new people into the org chart, where you had a company that was used to operating a certain way, caused confusion, and that creates frustration. And you know, it would have been nice to have resolved that sooner. I just don’t know [laugh] how you get the information to resolve it sooner, until you get some experience working together.

Brian: Right, right. Well, I think even knowing that those were challenging points, for the next one, you can say this is probably what’s going to be a mess—

Adam: Yeah, yeah, yeah. Good point.

Brian: —and I know from past, and there’s nothing we could have known perfectly ahead of time to get around it.

Adam: You at least know where the possible minefields are, so yeah, that’s a good point.

Brian: The known unknowns [laugh].

Adam: You’re making feel better already. Yeah.

Brian: Well, Adam, this has been great. Where can people get in touch with you if they want to say hi or for whatever other reason, to learn about Predictive Index?

Adam: Yeah, I am. Adam Berke across most things. And that’s B-E-R-K-E, so you can find me on LinkedIn is a good place. I’ll just stick with that. That’s probably, for business purposes, the best social media source.

Brian: Cool, excellent. Well, thanks for coming on Experiencing Data and chatting. This has been really informative. So, thank you.

Adam: Yeah, thanks for having me, Brian.