As product managers, we make a wide range of decisions on a daily basis.  These judgements vary from strategic decisions, such as feature prioritisation, to tactical decisions such as the font size in a new user registration form.  The difficulty in making multiple decisions is the mental effort entailed in the process while we switch from strategic choices, to the pixels in an immediate feature.  To ease up a bit on our mental effort, we can use data or our gut to drive our decision making process.  Or, we can use both.

My sister Orit Shaer, a computer science professor at Wellesley college, recommended watching a fascinating video of an interview between Marissa Mayer, CEO of Yahoo and Terry Winograd, Professor Emeritus, Stanford University.  The conversation took place at the HCI (Human Computer Interface) conference. During the talk, Marissa shared that in her early days as a product manager in Google, she was asked to provide more data to the company, rather than more opinions. In retrospect, Marissa says that she may have taken that too literally.  She alludes to a balance between data and one’s guttural opinion being the best way to move forward with product development.  I recommend watching the 3 min questions about data vs intuition starting from the 20th minute.

 

Learning from the experience of Marissa, it’s clear that product innovation needs a mix of both gut and data. “What is needed is a combination of gut feel (a hypothesis based on deep knowledge of the customer) and validation through well-constructed research that is precisely designed to test that hypothesis” writes Janet Muto, Managing Partner at Highstart Group. The tricky part is the validation process. In most cases, data can be manipulated to show anyone what they want to see.  When we consider Big Data, that assumption goes even deeper- ultimately showing different results to support a hypothesis or negate it.  It takes gut feeling, stemmed from business experience, in order to articulate unambiguous data-driven results.

B2B example

For example, let’s say you develop a B2B solution for SMBs and Enterprise companies. Looking at the last 30 days of user analytics, you can see that the “export to data warehouse” feature was only used by 30% of your customers. Based on the data itself, one may think the feature is not popular.  And perhaps, you could kill this minority feature?

But the gut feeling should tell an experienced product manager to dig deeper before drawing any conclusions.  Typically in B2B business, you segment your customers based on the customer type (SMB/Enterprise). In the case of our example, if you filter/group the results by customer type you will find a more meaningful answer. Among the organizations who use the “export to DWH” feature,  75% are enterprise customers the and 25% are SMBs. If the company’s strategy is focused on growing usage of enterprise customers, then the “export to DWH” is probably a star feature as it is used by more than 50% of the enterprise customers (your target customer).  As you can see from this illustration, data must be evaluated from multiple angles with an analytical and business savvy mind, in order to extract the best insights.

“In data science, intuition and analytics work together in tandem, each informing the other. First, intuition guides analytics. Second, analytics informs intuition.”  Steve Hillion, co-founder of Alpine Data Labs

Charity Water

Another example of navigating the paradox, can be seen from a B2C perspective as well.  Deepa Subramaniam, the director of product at Charity:Water, dealt with this judgement call in a major way.  Charity: Water is a nonprofit company, on a mission to bring clean drinking water to every person on the planet.  

At charity water, the company looks at the donation user journey as a funnel. Users who enter Charity Water’s website should ideally donate in a simple 2-3 step process. During brainstorming sessions on how to improve the conversion of the funnel (to optimize the number of donations), one member of the team suggested offering only one payment method rather than two (credit card and paypal). The finance team heard what the product team were discussing and immediately balked.  They were immediately afraid that the change would have a negative impact on their revenues from donations.  However, Paypal had a higher commission rate than the credit cards, so the product team hypothesized that removing the paypal option would both increase total donation dollars as well as increase funnel conversion rates.

After a few heated arguments, Subramaniam suggested to test the proposed optimization using an A/B test. 10% of the traffic of the website was directed to a different page where only credit card payments were available. After running the test for some time, the team found out that there was only a small drop in the number of donations while the donation amount increased by a neglectable sum. The final result was that there was no major difference when removing the paypal payment button. Interestingly enough, you can still find both payment options in the website today. I assume that it wasn’t worth the effort to kill the feature, or there were other reasons not mentioned.  By taking note of this process, it’s clear that there are a number of factors both analytical and personal, that product managers must take into account to develop a product or even kill a feature.  

Lessons Learned

As can be learned from the stories above, and many more in the field, working with data takes an immense amount of time and expertise.  Making data-driven decisions is a great way to make informed judgements, but it must be balanced with the soft skills and expertise of those who deeply understand the customer and the business at hand.

When you, as a product manager, invest your time working with data – you have to weigh the opportunity cost of not spending your time interacting with customers or innovating new ideas.  When working with data, the context  is usually used for process optimization such as conversion improvement, A/B testing and other experiments. But the real innovation, the “10X”,  may not come from optimizing existing data. The 10X innovation comes when you are in the same room (could also be a virtual room) with your customers or colleagues and brainstorm together on their real problems and needs.

When encountering the conflict of data vs gut, I suggest asking a few questions:

1. What is the problem we are trying to solve ?
2. Which segment of the users/customers is affected?
3. What is the desired result? What is the undesired result?
4. Try and map the different thresholds of the metric and decide what actions you will take in each case.

Product managers are decision makers. Good product managers aim to become better at decision making by using both their gut and data. Having business knowledge, and your customers in mind when working with data is essential to make the right decisions. You can improve by asking more questions that will lead you to the level of clarity that is needed to make the right choices of building the excellent road ahead.

 

Photo credit: Biarri Commercial Mathematics