S2 Ep. 4: How Have You Been Burned by Risky Assumptions?
Subscribe: Apple Podcasts | Google Podcasts | Spotify | RSS
Hope Gurion: Product leaders and teams are continuously walking a tight rope between urgency to meet customer needs that create value for their companies and taking sufficient time to ensure they don’t build the wrong thing. In fast-paced product decision-making cycles, assumptions are frequently made, even implicitly, that come back to haunt those product teams later. In this episode of Fearless Product Leadership, I asked 5 seasoned product leaders to answer the question “How have you been burned by risky assumptions?”
Welcome to the Fearless Product Leadership podcast. This is the show for new product leaders seeking to increase their confidence and competence. In every episode I ask experienced and thoughtful product leaders to share their strategies and tactics that have helped them tackle a tough responsibility of the product leader role. I love helping emerging product leaders shorten their learning curves to expedite their professional success with great products, teams and stakeholder relationships. I’m your host and CEO of Fearless Product, Hope Gurion.
Assumptions that never made the light of day frequently unsettle even the most customer-centric, evidence-based, hypothesis-driven product teams. They often surface too late by product teams eager to get products and features into customers’ hands quickly, expediting value-creation for their companies and not being stuck in analysis paralysis. In this episode and the following one, we’ll dig into the perils of risky assumptions—how they can disrupt teams, customers and companies and in the next episode, what techniques experienced product leaders use with their teams to avoid assumptions making asses out of, well you know.
In this first episode you’ll hear stories of product decisions gone wrong because of pesky, risky assumptions including:
Our Customers will pay for it
Our customers are ready for it
Our customers will accept our MVP--we can add that feature in a later iteration
We know enough to just go do it
Fearlessly tackling the question “How have you been burned by risky assumptions?” are:
Marc Abraham, Head of Product-Engagement at ASOS, former CPO at Settled
Rachel Obstler, VP of Product at PagerDuty, former VP of Product at Keynote Systems
Ezinne Udezue, VP of Product at Procore, former VP of Product Management, BazaarVoice
First, Marc Abraham of ASOS shares a story of how trying to deliver an MVP of a highly desired capability for customers backfired after assuming an important need for large customers could be differed in a later iteration.
Marc Abraham: Yes, thinking back to the time when I was burned by a risky assumption; what comes to mind is when I worked on a marketplace product effectively. A bit similar to had SC in the US; where we had sellers of artisanal products on one hand and buyers of those products on the other and I focused very much on the seller side of things together with the team. Whilst we were working on - what we did was creating a dashboard for the seller so they can see all their financial data and SKU data of all the individual products that they had with us on our platform in one single place - because up to that point that was pretty all over the place and we had lots of requests from our sellers saying, “Can you make it much easier for me to navigate, so I know how much stock I've got, how I'm doing on your platform, where my sales are coming from, all that good stuff?” and as we were doing our discovery and talking to lots of sellers doing demos, presenting early stages from pieces of paper to clickable prototypes; there were one or two sticking slightly bigger sellers. So, people who generate a million plus per year through our platform, mentioning something about access to all that financial and commercial data and being able to restrict the access within their company, so that not everyone can see it.
I heard that and I said, “Well yeah, now I take down board, but it might not be in version one of this dashboard that we're releasing” because obviously, we want to keep it very simple. The first mission that we have with this product is to get everything into one place and then we can start building on that, once people start using it and we can start looking in more depth at things like security, like usability, adding extra features, true MVP style. So, very clearly, what happened when we actually launched it, so very excited obviously, we've done it very quickly. No more than a month of kind of discovery and development time. So, we were quite pleased with ourselves as a team, thinking we got this over the line and I think it was only literally ten minutes after we just presented, that I can see literally from one end I've got the person heading up the seller proposition side of things coming to me from one end and our COO coming from the other end and both with very stern faces. What happened was that a couple of our really big sellers, those sellers that did a lot of business through our platform had complained, because they said, “Suddenly all my sales data is out in the open being (viewed by) everyone in my company” and these are businesses, not massive, but still good 10-15 people could access that data. They said, “No, that's not great, we don't want that, we want to keep a lid on that data. We normally don't share that data with our personnel because we'll be uncomfortable with what they might do with that data and if you don't fix that right now, I'm off your platform effectively.”
So, I think it's a good example of an assumption - well it wasn't an assumption; my assumption was when I heard initially as part in the discovery phase people saying, “Yeah, wouldn't it be nice to have some security and some kind of restrictions around the sharing of the data?” How big can they be right? Most of these companies are small operations, they know everything about anything anyway. I've only heard it a few times surely it's no biggie, let's release without and you know we can always fix it later and I've definitely learned from that experience, that even if it's one or two, it's worth spending some time. If anything, assessing the risk of not listening to that kind of requirement or that problem to solve rather than just making that very simple, gut feel kind of assumption to say, “Well surely, it's not a big problem” and to add to that, I think it also taught me by empathy; whilst I was thinking, “I've done my discovery, as you say I've listened to customers I feel their pain, but clearly, I hadn't necessarily spent enough time on certain things that they mentioned and so learning about empathy.
Hope Gurion: Next, Rachel Obstler of PagerDuty shares how she was burned by risky assumptions made by her and the leadership team about a product that was so obvious that they should just go build it.
Rachel Obstler: So, I think the most insidious or riskiest assumptions are the ones that that you don't know that you're making. What I have found is that a lot of times, those risky assumptions come from order taking, the bane of product management is order taking and that could come from a customer. A customer says that they want X but I actually think that that's one that most of the product team and product managers - good product managers I've worked with - are very good; they get very good understanding when those are happening and how to you know combat those.
So, I think actually the riskiest types of assumptions and the ones that are the most insidious, are what I'll call the executive assumptions. That's when some executive - I know am one too cite but, I say this like the class of people, I can also do this - say something like, “We need to do X” or this is the specific one; so, an idea was floated at our executive leadership team and it's an idea that I had actually proposed. The feedback I got was, “Do it. You should just do it” and so I took that as an assumption of there was an agreement, that this was going to get done. So, we put together the plan around doing it. Now, this this change would have involved changing a SKU; that means changing pricing and now that we're a public company, there are things that you need to do when you do something like that. It changes things in the market, it changes perception, all those things. So, in any case, we put together this whole plan it around it and presented it to the CEO and she was like, “Well, I think we should try this first, or this first; or are you sure we should do this?” and so it was immediately apparent that even though it seemed like there was a decision made or everyone likes the idea, is that you should just do it. That is different from actually putting together a plan and making a decision with all of your evidence and really signing off on it.
So, even when like these things sound like they're done or that then there is a decision. I think you always have to assume that if someone says we should do X, that you still own the decision, the assumptions, the reasoning. Never take it on face value that someone else knows more than you do because they hardly ever do and do all of your homework for it. So, that was one example of a risky I call it risky but it's really just a it was a mistake in assuming something that should not have been assumed.
Hope Gurion: Got it. Just to follow up, in that situation did you find that you had to go back and do it again? Like the thing you essentially lost, it seemed like you were moving quickly and then you ended up having to spend the same amount of time, as if it hadn't been green-lighted and you had to do it or it was there something else that you had to do as a follow-up to that?
Rachel Obstler: Yes, that is exactly what happened, was that we put together this plan as if it was greenlit and then we had to go refactor the plan to put together a plan and to get to the point where you could greenlight it, because we had the evidence that we needed at that point. So, we turned it into what it should have been from the beginning; which was, assuming we think this is a good idea, how do we validate that this is a good idea? Let's run this experiment, let's validate it and then let's go forward with the proposal right or the decision. So, it basically it set us back a couple weeks, because we should have been working on the experiments and the plan, as opposed to just putting together like this is what we think we should do without gathering the data.
Hope Gurion: Well, it's good that you have a CEO that values that and not just, “Oh good get it done faster” so that you did ultimately make all the right decisions and get all the right information you need it for your plan.
Rachel Obstler: Yes. Actually, was fine and it didn't cause any big issues, but I think both myself and my manager felt like idiots. We of all people should have known better that even though there was this meeting, where everyone said, “Just do it” that wasn’t the answer and they also didn't have enough evidence to say that and we still owned the decision and evidence gathering and whether or not we were ready to make the call. It was just a really good reminder to like these things happen a lot and you get so good as a product manager said, at recognizing them in customers when they're happening, but you have to also get really good at recognizing them internally, no matter what.
Hope Gurion: Now, Nicole Brolan, CPO of SEEK, shares the time she was burned by risky assumptions bringing a new product to existing B2B customers without really knowing whether there was enough value to justify paying for it.
Nicole Brolan: I think for me, we have an organization that has pretty heavy involvement from our strategy team and at the time that I can reflect on where we got pretty burned by a risky assumption, was where we had a feeling - we work in recruitment - we had a feeling that we could provide a service that customers were going to pay a premium for, because we were going to be able to help them further down into shortlisting. So, basically, they'd be able to give us a brief and we would be able to present them with a shortlist of the candidates and they've had to do nothing else. I think that the problem is that there were risky assumptions threaded in through there that the customers saw the value of that, in and above actually using recruiters and that there was going to be enough value in that, that people were going to say, “Yes I want to pay for it” and so what happened is we got too far down the path, without actually testing; is this viable? Is this desirable by people? We had a fantastic product manager, they pulled it apart and really kind of understood that actually this didn't have legs. It didn't have legs because customers did not see enough of what we were offering as it stood. The problem is because we've got so far down the line with a specific solution, the business felt that they couldn't wait for us to do the right amount of work to find a really desirable solution.
So, unfortunately what ended up happening, is we had to park the whole concept; which is a shame because I do think there was something in it, but it really taught me and actually began my personal journey for advocating that we really start with problems and hypotheses first and we don't start being to solution specific upfront.
I think what happens on the willingness to pay and I think this is what happened to us; people were like, “Well a job add costs three hundred and thirty dollars, a recruiter costs depending on the role, like twenty grand” and so they were like, “There's definitely value that we can deliver that people are going to be willing to pay more for” and it's like you know just because they're paying twenty grand doesn't mean that they're just going to throw money at you for delivery. If you just bundled a lot of things in, it doesn't work like that and so I think that was an important lesson for us as a business.
Hope Gurion: Ezinne Udueze of Procore shares how risky assumptions about customer expectations and anchoring caught her and her product team off guard when rolling out a self-serve solution.
Ezinne Udueze: There definitely is the time that I got burned because I didn't really pay attention to the risky assumption. So, we spend quite a bit of time really thinking through risky assumptions, not just as it pertained to the customer, but also for the organization.
Here is a story: My team and I were tasked with working on a self-serve model for one of our products and it's because we understood that it was taking the company way too long to onboard and implement our client. So, we figured if we gave them tools that would allow them to self-serve and on board, it would be better, it would reduce our overhead as a whole but more importantly, they could do it at their own time and speed and we were really excited about this opportunity. However, the sales process for enterprises always in commercial size companies, is a little more hands-on, it's pretty white glove and what we discovered was, even after delivering this self-serve onboarding and provisioning process, the reality is that the customer’s first encounter with us is very high touch. We were calling them and following up with them and when the sale was completed, we were leaving them in this self-serve experience.
So, yes, we ended up trading dissatisfaction with the time it took to onboard because there was a lot of back and forth in the original format for the product. We traded it all for that shock that the consumer had, our client has after they signed with us; because they no longer had someone to call on, or at least they didn't feel that there was somebody checking on, them because it had been designed to allow for them to take care of their own path and it just was a mismatch. So, on my part and the part of my team, I think it was we asked certain questions about our interaction with a client but we didn't step back and look at the assumption for the entire ecosystem, the entire journey and we thought number one, that the organization was staffed to handle and intersect with the product properly; but more importantly that our clients or a new client was ready to make that mental shift from sales into truly on boarding themselves. So, it's a big part of how we look at what we how we look and design for our new projects within product.
Hope Gurion: Next, Preston Smalley of Comcast shares a story of how understanding customers’ needs and pain points, is not enough if you underestimate the willingness to pay for of medicine if they have a good enough home remedy. And he shares knowing what he knows now, how he’d have tested whether those risky assumptions were indeed true if he could go back in time.
Preston Smalley: As I think back on risky assumptions, I can think of one particular time in particular early in my product career, that really burned me, and I didn't recognize it at the time what assumptions we were making. So, let me tell you a little bit of a how we got into that situation and what I've learned about it.
This was back when I was is still a general for a company called Plaxo and the whole intention was to try and get your contact list on your phone and your computer organized and one of the things that we heard from our customers, was that this was too much work for them, that they would have outdated contacts and they didn't know what people's latest phone numbers were and all this. So, we said, “Well look we can help you with that” and so we started thinking about some solutions, using machine learning and the latest Cassandra databases and we had artificial intelligence was going to go out and look across the internet and figure out some of these things that would help you solve this problem. Along the way we did user research we talked to people and we said, “Look are these problems that you've experienced?” and they were like, “Yes, absolutely! I'd love to have somebody that solved these problems for me” and we went ahead built the product, release the product and what did we hear? Crickets. Nobody was buying this product. It was it was a product that cost about five dollars a month and we started a probe as to what happened, we got into this situation where we realized people did have this problem, but they it wasn't worth enough to pay to solve it; that the pain that it created in their lives was easily solved. So, maybe I didn't have a phone number, but I could go hit somebody up on Facebook and within a couple minutes, I'd find the phone number.
So, it got us into what sometimes people talk about is this medicine versus vitamins problem; where it wasn't worth the medicine for them, it could be a little healthier for them but it wasn't worthwhile and I think it was in hindsight recognizing that in this product it was a big opportunity it was something nobody had really done before in the product space but it wasn't something that was needed enough by customers and we didn't recognize that, that was a leap of faith that we were making. This was when I came to really the realization that Lean Startup and some of those methodologies would be useful, because you could easily have solved for some of this using an MVP landing page test as an example. We can go see how it looks, “Here, we got this product, it's five dollars a month do you be interested in being part of our beta program or trying it out?” We would have easily flushed out some of these problems. I think you could have you could have faked a lot of that big tech that we were doing and Wizard of Oz style demo behind the scenes and really try and understand that.
Then there's another question that somebody raised around - I think this was - I actually just saw this last week the land of school guys were talking about how you could say, how disappointed would you be if this product went away. So, really trying to understand, if we would have given it away for free to people, we would have recognized that it would have been like, “Yeah it would have been okay if I lost it, but I probably would be fine” and so I think it me that really failure is just part of the process and that you need to really look hard at assumptions you're making or even fact that you think you have from research and really question what do you really know about this. In our case, we knew that they had this problem but do we know that they really wanted to pay for a solution and assessing those two things apart I think it's really important for a product manager to figure out.
Hope Gurion: Product leaders and teams are human. And we humans we are prone to all kinds of wishful thinking and biases. Eager to deliver value to customers, stakeholders and investors, we can be walking down that road paved with good intentions and full of overconfidence and optimism bias. However, experienced product leaders who have been around the block know that more often than we’d care to admit, it is the lack of surfacing these assumptions that bites us in the ass. We avoid delays of rework and the awkward dance of rebuilding confidence with customers and stakeholders after a failed release when we take the time to bring our risky assumptions into the light of day and experiment to resolve them.
In the next episode of Fearless Product Leadership, we’ll get into the techniques experienced product leaders use to help product teams and stakeholders realize their unspoken assumptions. And we’ll cover how to put those assumptions on the examination table and subject them to experimental rigor to sort the true and easy to manage from the false and possibly catastrophic.
If you’re a new product leader feeling like risky assumptions are about to burn you, I’d love to be of help. Contact me on Linkedin or Twitter or schedule an initial consultation with me using the Contact Me page at fearless-product.com.