Bad Customer Development Questions and How to Avoid My Mistakes

Summary: Although Customer Development can give us tremendous insight into market problems, it takes a lot of time – time that’s wasted if we do it incorrectly. Worse yet, poorly worded questions can cause us to reach wrong conclusions about what people want. The best questions don’t require customers to speculate about their behavior. Here I share real examples my bad questions and mistakes and offer some better alternatives.

If you’re starting Customer Development you’re getting ready to talk to a lot of potential customers. You started with an an idea, wrote down your key assumptions, and started flipping through contacts to see who you can interview.

Awesome! You’re off to a great start. Talking face-to-face with customers brings us insight we can’t get from surveys and clicks.

Unfortunately, conducting face-to-face customer development interviews is a skill that takes practice. I’ve been doing it for 5 years and I’m still learning. So many times I’ve asked the wrong questions and later realized I was wasting time or – worse yet – coming to incorrect conclusions and building the wrong products.

“Learn Nothing” Questions

I call “Learn Nothing” questions those that don’t result in any learning, just wasted time.

“What do you think of my idea?”

I LOVE talking about startup ideas. To quote my friend Patrick Smith, “talking about startups is entertainment like sports”. Fun, yes. Validated learning? No.  What do you think of my startup idea? is useless because if my idea is great people will like it. If it sucks, surely some people will still like it.

“If you could wave a magic wand…”

I know Steve Blank calls this the “IPO question”. I call it the “sit back and watch people ramble about things I have no chance of building” question. I’ve asked this question at the end of meetings and – after some funny looks – watched somebody ramble. It just hasn’t worked for me.

“Can you tell me about your problems with medical bills?”

I asked this question a few times before realizing that asking people to talk about problems just results in venting and no learning. This is particularly true with complex, personal, emotionally charged problems like those in health care.  “Well there was this one time….and then…but what really got me mad…”

“False Positive” Questions

Wrong conclusions are worse than no learning at all, and I call “False Positive” questions those designed to get customers to tell us what we want to hear.

All of us want our product ideas to be right – I want it, you want it, Steve Jobs wanted it. While our enthusiasm is our biggest asset, it is dangerous in customer development because most people don’t want to disappoint us.

“I’m building a product to help people manage medical bills. Can you tell me…”

Any question that starts with the solution already biases people’s expectations. Someone hearing this question might focus on a minor problem they had with medical billing in the past and convince us that it is a major source of pain.

Better: Skip the “I’m building a…” intro.

“How do you reconcile your HSA account with your bills, receipts, and statements to make sure you’re optimizing future tax savings?”

The “How do you do something complex to achieve results” questions can put people on the defensive. You can imagine someone thinking, “You mean I’m supposed to be doing something with that paperwork to save money on taxes? Oh no, I’m such an idiot, what am I doing wrong?”

Of course customers will be biased to tell us they need help with something after we cause them to doubt their own competence.

Better: “What do you do with that HSA paperwork?”

“Would you take a picture of a medical bill with your iPhone?”

This question sets up the prospective customer to tell us what we obviously want to hear.

Better questions would give us insight into how they currently work and whether the extra steps and inconvenience of using our products gives a promise of a big return.

Better: “Do you scan or file medical bills you get in the mail?”

This question is better because it gives someone the opportunity to disappoint us without realizing it, what Rob Fitzpatrick calls the “mom test”.

In this case, suppose your key assumption is that managing bills via in iPhone app saves time over filing or scanning. Should they respond, “Scan them? I’m too busy for that, I don’t even open them,” you’ve just invalidated your whole company vision with one question.

The Key to Effective Customer Development

In retrospect, the root of my bad questions was that I only had an idea – I didn’t have specific hypothesis I was trying to validate. I tried to replace the hard work of documenting and testing assumptions with meetings and simply wasted time.

Always remember that Customer Development is a big commitment that takes a huge amount of time, time best spent on something else if you’re not doing it right.

Before you line up a bunch of meetings and spend weeks talking to people, take some time to carefully consider what you’re going to ask and why.  The key is to make sure your questions are designed to test your assumptions.  I write them down and you should too.  

Talk with your co-founders or advisers about your interview script. If you need help, I’m happy to review it with you.

Photo credit: anyjazz65

13 Comments

  1. Ray January 22, 2013 at 12:05 am #

    Hi Kevin! Wanted to add that customer interview is more an art than science.

    Really like how you pointed out that healthcare is a sensitive area, and the way one conducts effective interview can be vastly different from working with another segment.

    I believe the key is to also consistently review the effectiveness of the questions, write them down, compare the results, and tweak them to find out the best way to gather information from the particular early adopter group.

    Always Be Testing!

    • kevindewalt January 22, 2013 at 1:13 am #

      Definitely agree Ray! There is no “one size fits all approach”.

      I believe the key is to also consistently review the effectiveness of the questions, write them down, compare the results, and tweak them to find out the best way to gather information from the particular early adopter group.

      Well said, better than my post actually. I imagine you’re one of the world’s experts in this process having seen so much at LSM.

  2. Prakhyat January 22, 2013 at 12:51 am #

    Kevin,

    Great advice as usual. You mention that time commitment for customer development is huge as well as a big commitment and I absolutely agree with it, but isn’t it a necessity as well. I mean what is the use of developing something when you are not even sure of it being getting accepted by your customers. How do you find this balance and what process should then be followed for the Customer Development that actually helps?

    • kevindewalt January 22, 2013 at 4:04 am #

      Hey Prakhyat,

      The bottom line is that you want to avoid the biggest source of waste: building something or otherwise spending money on time on something that nobody wants. Often the fastest way to do this – especially when you’re just getting started – is face-to-face interviews with prospective customers to validate whether your assumptions are right or wrong. Done correctly it yields tremendous insights. Done incorrectly and its a waste of time or, worse yet, misleading.

      I don’t have space here to layout the whole process – you can see Steve Blank’s Udacity Course or read Ash Maurya’s Running Lean for that. Or simply start by writing down your key assumptions, good questions to ask, and then start testing them with face-to-face interviews.

      If you can’t do face-to-face interviews then you’ll have to find another process such as testing through content, Google Adwords campaigns, or landing pages. Hope it helps!

  3. Randy Whitcroft January 24, 2013 at 5:07 am #

    Kevin,
    This is a great post. It is a key element to customer development in that it is not just about asking questions, but asking the right questions. Although there are the many different questions that you can ask, you need to have the core desire to avoid the costly mistake of building something that people do not want. You have to be fully committed to the principles of the Customer Development philosophy and methodology to get the benefit from it. By being fully committed to this, then you are much more likely to ask the right questions. Seems simple…but in my experience this is the biggest hurdle to allowing people to get the most from the Customer Development process. The examples that you have given around the magic wand, false positive, and leading questions are good examples of this….don’t ask questions to hear what you want to hear….ask questions that allow you to hear what you need to hear to save yourself a great deal of precious time, money and energy.
    Thanks for sharing this Kevin.

    • kevindewalt January 24, 2013 at 7:22 am #

      “Seems simple…but in my experience this is the biggest hurdle..”

      It definitely has been for me. Really the hardest part has been changing my mindset, getting myself to think in terms of discovery. So simple in principle…so hard in practice.

  4. David Telleen-Lawton February 8, 2013 at 2:28 pm #

    Kevin:

    Your examples are great for making the point. Your specifics make it come alive.

    I’ve done customer discovery for 25 years…in the trenches, setting the meetings, coaching on the presentations, leading the teams into the meetings, debriefing afterwards, rejiggering the questions and presentation, pivoting, deciding on the next wave of meetings, etc.

    Here’s how I boil down the key elements:
    * Specificity – Just as you say, have a specific hypothesis…also, if you have a product/service in mind, be specific about what it does. That’s where screen-shots or other means to make the product tangible come in. “It’s this size.” In the old days, they used to say, “Garbage in, Garbage out” referring to poor programming. I say, with reference to customer discovery meetings “Garbage out – hey, this will save you millions of dollars, then Garbage in – okay, then I’ll buy twenty.”

    * Ask for operational FACTS before asking about product/benefit/feature OPINION — you have some examples where you do ask for facts, but then you added on your “result” phraseology and and you saw, it completely kills the focus of the question. “Tell me about your HSA.”, is a great start. “Do you recall getting a statement?” “When you got the statement, what did you do with it.” (note: not what will you do with it.)

    * Invoke the selling model. Not the selling as in bludgeon them with details and features. Selling in the sense of closing on a commitment. Even if you think you know what they will say, ask them anyway. “Based on your situation, the problems we claim we will solve, and your current thinking, what is your current thinking on being one of our first customers?” Bite your tongue. Whether you hear something positive or negative, if it doesn’t comport with the operational facts you heard earlier, drill down and understand what’s missing. It should fit, so if it doesn’t, one side or the other is missing some “facts”.

    * Process in real time. When you hear something that might affect a feature, performance, frequency of use, etc., ask more questions to understand…to understand why they said that, to understand why that’s important.

    * I wonder if you went out alone or with someone else. If you have a teammate, taking notes, with whom to discuss what you learned, catching exact quotes, then it’s very hard to keep hearing fluff and feeling good about it.

    * Lastly, you should be putting each interview as a row in a spreadsheet with columns being the key attributes/characteristics/answers. At first, you don’t know which subset is key, so you have many more columns than you will eventually need. Besides some demographics there should be a rating on need, likeliness of purchasing, what they’d use it for, the good news, the bad news, the quote of the day.

    There is an art to questioning, but there’s a whole lot of good technique, too.

    • kevindewalt February 8, 2013 at 6:15 pm #

      David,

      Awesome comments, you obviously have a great deal more experience than I do. I don’t disagree with any of your points.

      WRT your questions:

      “I wonder if you went out alone or with someone else”: Currently I’m doing mine alone, mostly because I’m a single founder at this point. I then try to roll-up what I’ve learned and present to my advisors at our periodic pivot-or-persist meetings. I can definitely see the advantages of doing it with others, I guess I’ve never found a good way to do it logistically in the past. Especially for enterprise customer meetings I think this would be critical.

      “Interview in a row”: I’ve done that as well, although recently I’ve actually switched to MS Word. I do an interview script for each person and then type out by hand a summary afterward.

      Great thoughts.

      BTW, have you watched this Steve Blank video? Both funny and with great insight. http://startupweekend.wistia.com/medias/tao3s8hf7l

  5. David Telleen-Lawton February 12, 2013 at 2:53 pm #

    Kevin:
    This is absolutely the best post I’ve ever seen for helping teams understand the problem of asking the wrong questions. And I agree on your “IPO questions” comment heartily.

    After 25 years of market validation (or customer discovery as it is known today), I disagree with one of your commenters that it is more art than science. I say it’s more science than art and here’s why I write that.
    Notice that most (but not all!) of your questions are asking for opinion. By asking for operational fact before purchase (or other future action) opinion, one collects important foundational information. This data is then used to see patterns to help you qualify better, craft marketing messages, size your market, etc.

    For example using medical bills, “When was the last time you received a medical bill?” “Did it come from your insurer or your provider?” “Thinking about that last medical bill, tell me about it’s path from your mailbox to it’s final resting place….” “Really and why did it sit on your desk for four weeks?” , etc. Notice the processing in real time…you are trying to understand the decision/action process of your prospect.

    The best way I have found to see these patterns (in conjunction with copious notes and catching direct quotes) is to put them on a spreadsheet with each interview a row. Across the columns are markers of the meeting — both fact and opinion which are dictated by the particular customer type, etc.

    For example, I would imagine some column headings would be Gender, age, soc-eco status, married, # living at home, pays bills, last doctor visit, # of visits 2012, insurer, type of plan (hmo, ppo), etc.

    Once you get enough interviews, it will be fairly clear which “5″ characteristics tell you whether this customer will be a good (or bad) fit for your current solution.

    Similarly with the questions, you will not have to drill down as deeply in as many areas once you see redundant answers and thus are not learning anything by asking them. Of course the outliers will be golden.

    The other great advantage of learning operation facts before future-action opinion is that when you get to their opinion (“I would definitely buy this and I surely will recommend it to others.”) and it differs from operational fact (“No, I haven’t heard of those other solutions.” “Hmmm, I was very happy with that, but no, I didn’t let anyone know that. I really don’t know others in my field…I’m pretty isolated.”), then you can drill down to understand what experience they have (or you have) that makes this irrational claim rational. I have found over and over again, people are very rational…when they appear irrational it’s because of a difference in data (experience), not a difference in their regard for making rational decisions.

    Lastly, you DO need to keep the selling model in mind. Your are NOT doing an opinion survey. You are trying to determine your product-market fit and your sales model. You DO need to be “selling”, but not the caricature selling. Rather the questioning and trial closing.

    Just as Kevin states, “Skip the ‘I’m building a…’” statements (product-centric).

    However, I’d add skip the “I’m not trying to sell you anything statements.” Not only are they received similar to “I’m telling the truth, really.” type of statements, it is plain wrong-headed. Here’s what I say, “Thank you for agreeing to meet with us. As we told you on the phone, we’re working on helping to solve the problem of this or that. We’re trying to make sure our solution hits a bull’s-eye with the problem and understand it’s value to users. It’s also a chance for you to put your thumbprint on our solution to be sure it solves your problems. Our solution is still under development and after we learn a bit more about your situation and requirements in our discussion today, we will show you how far we’ve come. Although we won’t be asking your for a purchase order today, we would expect that if our product delivered value to you and fit with your circumstances, that we would be coming back here and asking you to purchase it. So please be very candid with your answers. How does that sound? Should we get started?”

    Now you’re not wasting anybody’s time!

    • kevindewalt February 12, 2013 at 5:38 pm #

      Great comment David, lots of awesome details. WRT “not selling anything”, I’ve heard this consistently from others as well and (without realizing it) haven’t been using it in my script either.

      On the spreadsheet, have you seen what Ray Wu is working on? I think he would be interested in your (obviously experienced and well-informed) opinions: http://blog.raywu.co/post/42414946417/3-steps-to-great-customer-development.

  6. Matías Hernández March 31, 2013 at 9:39 am #

    Hello everybody, just thank you all for the article and the incredible coments. I will be using some of your advice to write a post on my blog and help spanish entrepreneurs If you do not mind. First timers on Cust.dev incur on the se errors and the techniques you talk about are crucial to avoid losing time and stay focused on what really matters. Best regards.

  7. evershine March 23, 2014 at 1:55 pm #

    Kevin – really appreciated your insight at the NEXT Toronto event. We at LocaWoka would be surely asking for your advice! Cheers, Shabbir

    • kevindewalt March 23, 2014 at 6:46 pm #

      Anytime! Glad to be a part of it.

I read EVERY comment and want to hear from you