Q: How would you suggest we can influence data-driven culture from bottom-up?

A: Bottom-up culture changes are hard but not impossible. I’ve had some success doing it myself and have seen other teams do this as well. Some tips:

• Aim for a meaningful and visible win. One that both management and peers look at and say “that was a great win, and it wasn’t by chance, we should learn from this”. 
• Find the right group-wide channels to communicate progress about that specific effort. Such channels include weekly meetings with a wide invite group, FB Groups for post-meetings and mid-week updates, slack channels (dislike them generally for lack of a reasonable threaded conversation implementation), or if you have to, emails… Communicate often, and put focus and data front and center. 

Re. data-driven:

• This is about teaching the org to make logical informed decisions based on market opportunities and signals. 
• Note that data-driven hypotheses are mostly relevant for growth efforts, i.e., optimizing existing, post-product-market fit, value propositions. 
• I like data-informed for 0->1 projects (pre p/m fit) since most such efforts often rely on different types of market signals like key customer feedback, and where data alone isn’t sufficient to describe the opportunity or confidence in it. 

Q: You mentioned that it is important to look at why a problem is important. Is it of value to also look at why it is not important (devil’s advocate)? And will the help avoid self-selection bias? 

A: Great question, and yes, of course. Training ourselves to come up with the greatest counter-arguments for our own hypotheses is a great way to grow and stay focused. As long as it’s genuinely well-intended and maintains intellectual integrity. 

Q: In the superhuman example – asking  “would you be upset if we take this away” – what should you do in cases where you have only several hundred users, how to re-ask them this over and over without “bugging “ them? 

A: I’ve seen a number of different ways to tackle small groups:

1. Ask a random sample group each time and rotate. 
2. High touch instead of survey, with a much smaller set, and rotate. I usually use this approach because it provides a lot more signals anyway and ~10 conversations will probably yield more signals about future retention potential than 30 surveyed using the above method. 

Q: Market discovery: “If I had asked people what they wanted, they would have said faster horses.” Ford. When should you override the market’s feedback? 

A: This quote is often misinterpreted, so thanks for this question. Potential clients often speak in the language of solutions, and often those suggestions are not reasonable or optimal. But if you listen well you can reverse engineer it back to the problem/opportunity. In this case, people want to commute faster. That’s a great opportunity with multiple strategies (trains, cars, Uber…). 

I recall a case where a person working for a big client of ours asking: “Can you please add a button to download the report to PDF?”. After looking closer into this, it became clear that she wanted to share the report with her peers, and since it couldn’t fit in a single screen she couldn’t just copy/paste it into an email. Our solution was to add a mailing list and send weekly updates to everyone on the list. For context, this was back in 2002, and today different approaches would probably work better. 

Q: Could you give an example of how this framework you talk about comes into play in the context of roadmap products?

A: Roadmaps themselves should start with an opportunity context. In most cases it’s a continuation of existing business opportunity (repeating why it’s important, where to focus next, and how to measure success), exploring a new one, or providing a “theme” for the upcoming quarter that feels more tactical (e.g. time to clean bugs). The derivative of that overall strategy is a list of products. Features, activities, and other efforts that will deliver those business goals.