Using a Smoke Test

Note: this is a series on making a digital health startup.  The first post is here.

Last week we talked about things that don’t matter in starting a startup.  Today we’ll discuss a quick method to find out whether people really like your idea.

In the past we’ve discussed the benefits of surveys in testing your idea.  A well-designed survey can help you know what people care about, and it can give you a decent insight into the respondent’s desire for your product.  You can ask a bunch of questions relatively efficiently, and you can adjust them over time to really understand your population.

Better than a survey

But now let’s say you’ve run a few surveys, decided what you want to make, know your features, and know the language you want to use.  Now you want to know: will people buy this exact product?

In a previous survey you’ve probably already described the product you want to make, and you’ve probably asked if people would buy it if it were available.  But during a survey people sometimes suspend disbelief, and you can’t always trust their responses.  My friend Tivan, an expert in surveys, says he reduces survey results by a factor of three to estimate a product’s real demand.  If 60% of people say they’ll buy your product, reduce that 60% to 20% for real life, he says.

This is a good, conservative approach, but why not get a real number? For these cases, Tivan suggests running a smoke test.  Smoke tests are actual websites, with actual products, and actual prices, and an actual “Buy Now” button.  The only difference is, at the end of the “Buy Now” option you don’t end up with a product but a pop-up that says, “Thank you so much!  Our product is still being tested, but we’ll let you know when it’s ready to be sold.”

If someone sees a description of your product, a photo, and a price, and she still decides to click “Buy Now,” that’s a pretty good estimation that they actually want your product.  Of course it’s not the same as cash, but it’s a level higher in accuracy than a survey, and it should provide further comfort that you’re making a product people care about.

Building and doing a smoke test

So how do you make a test like this?  Making this test is just like making the survey you built earlier, but now instead of 20 questions, there’s only one question (the “question” is the product’s description, and the “next” button is “buy now.”)  There are many sites that can help, but my recommendation is Typeform (or another web-based form builder, but really Typeform is best).

Typeform allows for beautiful product pages with zero coding.  Make a simple, one question survey, and make sure the button says “buy now.”  Here’s one example I built in 90 seconds:

Screen Shot 2014-10-31 at 8.46.58 AM

If someone visits this site and clicks the “buy now” button, I guess she could be lying, but it makes me feel pretty confident she wants the product.

Steps I took:

  1. Started a new typeform
  2. Clicked on the “Welcome screen” question on the left
  3. Found an image of a toothbrush on Google, saved image to my desktop, uploaded it to the question
  4. Wrote a description of the toothbrush and added a price
  5. Changed the text of the button to “buy now”
  6. Saved this question
  7. Added a “thank you” screen via the “statement” button on the left, added a “thank you” screen via the “thank you” button on the left

That’s it.  Obviously there are ways to make this more complicated:  You can use a better image rather than one from Google (but why?  Surely you can find an image that’s a decent approximation of the thing you’re making, even if it’s an app).  You can add a payment form within Typeform and actually sell the product (but then you need a PRO plan on Typeform and anyway then you have to deal with receiving cash and the potential ethical implications).

Or you can make a fancier description.  But why bother?  You’re just trying to use a quick & dirty approach to determine whether people want your product…and if you can build a website in 90 seconds why do anything else?

Product description

You might think your product has a million features and needs a long description — and how could anyone possibly decide to buy your product without a feature-by-feature breakdown — but I suspect you can find a short description for your product.  “Brevity is the soul of wit,” they say.  Focus on just the most important things, the most important 20 words.

Or if you cannot possibly limit to 20 words, switch the text of your “buy now” button to “learn more”, and then use another question to provide the full description before you ask them to buy.  But again, probably overkill.  If you’re making something awesome, write out the most awesome feature in a single, declarative sentence, and put it out there. You can always adjust the sentence in a future test if the results of version 1 aren’t positive.

Choosing your survey respondents

There are lots of approaches to take in terms of your survey population, and each is biased, so your best bet is probably to do the test a few times, with different populations.  Let’s say you’re making a product for a general population.  Here’s one approach to take:

  • First, post to your friends on Facebook.  This is a nice broad sample (and obviously very easy to execute), but it’s likely biased towards people with some similarity to you.  I would recommend starting here because it’s easy, and then you can use the results from this survey to adjust your next one.
  • Next, email your 5 closest friends and ask them to post on their Facebook walls.  Again, similar potential biases, but this time you’re going out with a broader sample so theoretically the bias is lower.
  • Finally, use google adwords.  This is potentially a more random sample, but then you have to pay to set up adwords (adds another hour at least to your experiment setup time).  So only do this once you’ve done the other approaches.

But sometimes you don’t want a random sample.  If you’re making a product exclusively for PhD electrical engineers, then polling your entire facebook wall is probably not a great idea.  But is there a facebook group for PhD EEs?  A mailing list?  Maybe start there.  Start with the people who are most likely to want your product and establish demand there, and then you can move out to a more broad sample.

In summary, use a form builder to build a simple, single question smoke test.  Design your test quickly, and run your survey quickly.

Next week we’ll talk about the volume of tests you should expect to run before you come upon an idea worthy of your energy.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s