Good Survey Questions

Note: this is a series on making a digital health startup.  The first post is here.

Last week we talked about deciphering what people really think when they give you feedback on your startup idea.

This week we’ll talk about asking good survey questions.  As I mentioned earlier, a well-designed survey is a good way to find out the extent to which people like your idea.

Does Someone Want Your Product?

There are many potential goals of a survey.  In my opinion, the most important goal is to find out whether someone wants your product.  I’ve found it’s best to be direct, descriptive, and action-oriented.  A good question is: “We are developing an app for cancer patients.  It provides information on treatment, a chatting service for you and your physician, and an automatic prescription reminder service.  It costs $2.99.  If this product were available today, would you go on the Apple App Store today, buy it, and agree to use it for two weeks?”

Why is this a good question?

  • First, it’s direct.  Instead of saying, “We are considering making…” or “We’re thinking of making…”, it puts the survey respondent in the frame of mind of a real product that will exist, and hopefully the respondent will be more likely to give a truthful response.
  • Second, it’s descriptive.  If you describe your product in vague terms, it’s hard to know whether the product will be useful.
  • Third, it has a specific action item.  Asking someone to download an app, pay for it, and then use it for two weeks is a lot of work.  Only if someone really likes your idea will s/he agree to such demands.

But still this isn’t perfect.  People who say they’d say yes sometimes don’t actually follow through.  A friend of mine divides the results of responses like this by three; in other words, if 60% of people respond “yes,” he interprets that result as 20% will ultimately buy.

One way to counteract this is to use a survey service that allows credit card ordering (such as the wonderful Typeform (note: you should probably go visit Typeform right now just to see their homepage)) and let people place an order.  Let them join the “special early order group” for a 10% discount, and take their pre-order cash.  This way there is zero ambiguity about the intentions of someone who has already pre-ordered your product.

After you ask this question, you might want to ask a question about features.  “Why do you like this product?”  “Given this list of features, which one do you like most?”  “If the first version of this product had only this feature, would you still buy it?”  These questions help you establish your product feature roadmap, and they help you figure out what people really care about.  Is it one feature, or is it the price, or is it the combination of things that matters?  Understanding these nuances will help as you begin making the product.

Note: You might be worried about the ethical implications of telling people you’re actively building something that is (at the moment) just an idea.  I think a good way to address this is to get folks’ email addresses during the survey, and to send regular emails.  If you decide not to go forward with the concept, send an email explaining why.  If you’ve accepted pre-order money, refund it (with interest, if possible).  As long as you’re prompt and transparent, it’s hard to be faulted.  You can also offer to give the source code to anyone who’s interested in pushing forward with the idea, and you can list other places people can go to get similar services.  But really, it’s not a huge deal; people generally understand that sometimes popular projects are shelved, and in the mean time, you’ve gathered really useful data about your product.

Other Survey Questions

Another goal of surveys is demographic information; it’s important to understand the category of person you’re dealing with, for the purposes of qualifying the relevance of a person’s response.  Let’s say you’re making an app for cancer patients and you run a survey, and 100% of people say they love your idea.  Is that idea going to succeed?  Maybe, but it helps to know whether the survey respondents are cancer patients…or whether it’s your ten best friends filling out the form because they’re trying to be supportive.  For example, I recently ran a survey on a product for nursing moms, so I began the survey with the question: “Are you, or have you been, a nursing mom?”  It was interesting to see how the responses among both categories.

It’s also important to know demographic data so you can develop personas of your customers.  Maybe 100 people like your idea, but for 3 different reasons.  Using demographic data helps you divide those people into categories, to help you customize products to each group.

Another potential goal of a survey is quantifying the size or magnitude of the “problem.”  In the survey I ran for nursing moms, the problem we were seeking to address was mastitis, or painful inflammation of breast tissue.  We were thinking of designing a product to address mastitis, so we wanted to know the extent to which mastitis affected these moms.

In a different example, I wanted to know the biggest pain points involved in biking.  So I ran a quick survey.  The first was a demographic question, designed to qualify the respondent (I wanted to make sure the respondent was actually a semi-frequent cyclist).  Then I asked six questions with a 1-to-10 scale, one being “not an issue” and 10 being “bothers me all the time”.  “For the following questions, please tell us how much each bike commuting issue bothers you.”  1: I’m worried about my bike being stolen, 2: I hate biking in cold or rainy weather, etc.  I then put in a freeform answer at the end, to capture anything I might have missed.  “Is there another bike-commuting issue that’s really annoying for you?”  And then I asked 30 people to fill it out (you need enough people to fill out the survey that you feel comfortable that the results are real).

In this way I was able to see which issues were most painful.   Also, given two options, one with a higher average pain but bell distribution, and the other with a lower average pain but spiked distribution at the 8 or the 9, I’d prefer the second, because it means there is a core of people who feel strongly about an issue, and those can be the early adopters.

Other General Tips

  • Try to keep respondents un-biased as long as possible.  Keep the title of the survey generic, so people don’t unconsciously tailor their responses to what they think you want.
  • Keep it short.  Every question is another opportunity for people to give up.  Only ask what’s absolutely necessary.
  • Know what you want to get out of every question.  Before you ask a question, write out on a sheet of paper the thesis you’re trying to test.
  • Use free-form answers when you want a broad number of responses, and then switch to structure multiple-choice when you want to analyze the data quantitatively.
  • Show the survey to a few buddies before you release it, to work out the mistakes and unclear language you’ve probably accidentally included.
  • Continually tweak the survey as you learn things.  If no one likes your product at $10, lower the price to $5 and see what new people think.  If everyone likes it at $10, raise it to $15 and see if your percentages decrease.
  • End the survey with a, “Is there anything else you want to tell us,” because often people will write brilliant things you’d never think to ask.
  • See this blog post for further information, from a guy who’s had two successful Kickstarter campaigns.

Check out this survey I ran for nursing moms to get a sense of some potential techniques you can use.

Next week we’ll talk about things that don’t matter when starting a digital health startup.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s