The Power Of Small Experiments

By Alvin Ånestrand @ 2023-01-27T17:30 (+10)

This post is for discussing how to make really simple tests in order to spend as little resources as possible while still acquiring valuable feedback for what to focus on. It is inspired by ideas from the Lean Startup movement. There are a lot of ways to make tests that do not require a lot of money or time, for example when considering important choices regarding career paths, projects or what to study.

Note that I am not an expert in entrepreneurship. Also note that in the spirit of this post, this post is itself a small experiment. I forced myself to not take more than a few hours to write it, and if it gets enough attention or appreciation, I will know that it is worth writing more about it.

If you have advice regarding small experiments,  literature recommendations, or other thoughts or feedback on my post, please comment!

A few techniques

Pretotyping

Pretotyping is a very powerful technique. It is often used for testing product ideas. The basic idea is to, instead of making a product, test its potential without making it. There are many ways to do this, and I will cover a few of them here, and list examples. The person that came up with the concept, Alberto Savoia, wrote a short book about it, Pretotype It.

A few of my pretotype examples include tricking people. Temporarily. I don’t like dishonesty, and you should tell the truth afterward if you actually trick anyone.

• The Mechanical Turk: Instead of making an automated tool, do the thing the tool would do but manually. 
  • If you are thinking of designing a cool new app for recommending restaurants, tell others that you have already made it, have them test it, and just do what the app would do but manually and see how they like it. If it’s impossible to trick people that you already made it, just tell them straight up you will do it manually. It’s still a good experiment.
  • If you are thinking of buying an AI writing tool that costs a lot, just have a friend do what the tool would do for an hour.
• The Pinocchio: Instead of making a product, make a completely fake version and just imagine that it functions as it should.
  • Instead of designing a new type of ice cream machine, just imagine you are using it before eating ice cream every day for a few days. Take a bowl that you imagine to be the machine. Do you become tired of doing it? Is it fun? Does it take too much space in the freezer? You can figure things like that out before making a prototype.
  • When deciding whether to move, think hard about whether the day to day life in the new place would actually be better. Imagine living there. Suppose it is sunnier, will that actually affect your daily well-being? (I actually don’t think sunnyness is one of the more important aspects to consider.)
• The Provincial: Instead of launching something world-wide, start small.
  • Instead of starting a blog, and reading a lot about how to write and how to arouse interest, just write a couple of forum posts and see how it feels first. It might reach a smaller audience, but depending on the forum you will get very valuable feedback.
  • For an app, instead of learning how to make it for several platforms at once, do a simple version for only one platform. You might have to remake the app later in another programming language, but you can get feedback fast.
• The Minimum Viable Product (MVP): Make an early version of something (usually a product), with only the absolutely necessary features, and as basic as possible.
  • For the app example for The Provincial, start with making the app as simple as possible. If people are supposed to communicate with it, start with only allowing simple text messages.
  • If you are writing a non-fiction, think of what the most important topic is, and start by writing a much smaller book about what is most important, and get feedback on it.
• The Fake Door: Create an opportunity that is completely fake, in order to test how many would bite if there was an actual opportunity.
  • If you want to test how many would buy/download/read your creation, you can make an ad for it, and see how many clicks the ad. If the ad links to a website where people can sign up to be noticed when the opportunity exists, you can compare the ad clicks to the number of sign-ups. Remember to apologize for advertising something that does not exist.
  • If you want to start a project, ask people if they are interested in collaborating with you before starting it. You could tell them that you have already started it, or if you want to be more honest, tell them that you will start a certain date and then actually start at that date if you get a few people to join.
• The Pretend-to-Own: Borrow or rent instead of buying things for experimenting.
  • Instead of making a website from scratch, use an existing website creation tool to make a simple version. It might be free, it might cost a little. Use it for experimenting, perhaps for the ad test for The Fake Door, until you know it is worth spending time making a more professional website. You could also use a cheaper website tool first, and then upgrade if you realize it is worth it.
  • If you want to buy a computer/phone of a brand that you are not used to, borrow someone else’s for an hour.

Ranking options by several dimensions

Another useful technique is to simply rank options in several dimensions, and simply add the scores for every dimension. Do this for all options, and compare them. It sounds simple, but it is surprisingly powerful. I first read about this in Thinking, Fast and Slow by Daniel Kahneman, which is a must-read. Here are a few examples.

• When I helped my brother decide on a Bachelor's project, we decided to use these dimensions: Learning experience, tractability and feedback, expected group diversity and dynamics, positive impact (does someone profit from it directly?), autonomy and diversity
  • He noticed, after doing the exercise, that the project he first planned to apply for got a lesser score than another project that scored high in every dimension. He changed his decision. Any good test should be able to change decisions.
• In Thinking, Fast and Slow, one example was for predicting marital stability. “Frequency of lovemaking minus frequency of quarrels”. You don’t want the answer to be negative. Apparently marital stability is well predicted by this simple formula. 
• Another suggestion by Kahneman regards interviewing applicants for a position. Decide on about six dimensions (in this case traits) that are as independent as possible from each other, design questions for each trait, and think about good and bad answers to the questions. Avoid the halo effect, evaluate each trait one at a time, giving a score before you move on.

Kahneman suggests that a good rule of thumb is using about six dimensions.

This method can be used for any important decision including several options. The method is more accurate than simply ranking the options without specific dimensions in mind, and after you have ranked the options by your different dimensions, Kahneman explains that you will likely make a better decision even if you don’t add the scores. You will have gotten a more comprehensive and accurate view of all options. There are more good examples of using this method, and using algorithms in general, in his book.

Also remember that for some decisions it might not be a good idea to use this technique. When you reason about something, you might get a reasons-generated attitude change. It means your attitude changes to match your reasoning. This attitude change might be temporary, and if you make a decision with a temporary attitude you might regret it. For any decision where your attitude is the most important factor to you (perhaps relationships, or a new car), intuition might be a better tool for decision making.

The Mom Test

It can be hard to get honest answers from people. If you say, “Hey, I am thinking about doing [insert something here]. Do you think it is a good idea?” you will probably get something like “Oh I love that idea!” or “You should definitely try it out!”. People can be way too supportive. You want to know why it is not a good idea, if it indeed is not a good idea. However, pretotyping or designing small experiments might take longer than simply asking the right person. The solution is The Mom Test. The Mom Test is a book by Rob Fitzpatrick, but I will refer to The Mom Test as a technique.

The Mom Test is a technique for getting honest answers. Just do these three things when you are talking with someone:

• 1. Talk about their life, not your idea
• 2. Ask about specifics in their past, not about generals or the future
• 3. Talk less, listen more

To get honest answers, do not tell the person what you want advice for. Don’t ask leading questions. Give information when needed instead of giving suggestions, stating hypotheses or stating your opinions. A good rule of thumb is to talk 20%, listen 80%. I suggest reading The Mom Test for elaboration.

Career examples

If you consider a certain job, the commonly accepted test method is to work temporarily in that job. You could do it for a few months, or get an internship position, or something similar. But you can find out if you are a bad fit for the job way faster.

• If you want to test whether software development is a good fit for you:
  • Know how to code? Imagine / test doing debugging for a few hours
  • Don’t know how to code? Take a free online crash course! Preferably start with a really short one
• If you want to test whether teaching is a good fit for you:
  •  Try giving a lecture about something you know for some family members that don’t know as much on the subject (The Provincial)
  • Visualize a day as a teacher. How would you handle a really difficult situation? What if two children cannot shut up? (The Pinocchio)
• If you have decided on a few possible career paths, list a few dimensions for ranking them. Suggestions:
  • Positive impact (check 80,000Hours)
  • Job satisfaction
  • Required study time, given your current education
  • Wage
  • Personal fit (make a simple experiment!)
  • Check these job characteristics for more good dimensions

Good experiments and good results

A good experiment should be behavior changing! If you don’t expect an experiment to change your decisions, don’t do it.

Experimentation almost always beats asking for people’s opinions, even if you use The Mom Test, but talking with people is often faster.

Remember to design experiments that give as objective results as possible. If you can interpret everything to match your current preferences, you have failed.

Remember, any result is a good result. The results telling you that your idea is terrible are the most important. The harder a result is to face, the more important it is. You want to change direction as soon as possible if the idea isn’t good, or you will not be altruistic effectively. I also recommend reading Conservation of Expected Evidence, a LessWrong post by Eliezer Yudkowsky, to understand test results better.

Creating and executing tests

I find it useful to list assumptions and hypotheses regarding things that I want to test. A suggestion is to write a question for every assumption and hypothesis that you come up with. If you have an assumption that a project you want to do will take about three months, your question could be “How long does it normally take to do this type of project?” (Use the outside view!) Then rate every question by how important it is and how easy it is to design a test for it and find an answer. Then start with the questions that scored highest!

Now when you have your questions, use techniques like pretotyping or The Mom Test, and design tests for each question. Remember, REALLY SIMPLE TESTS. The best tests are the ones that take minutes. The tests should not take more than a few hours if it can be avoided. And I don’t mean the time until you get the test results, I mean the time it takes to set up the test. If you for example use The Fake Door with a website, you might have to wait for getting enough data, but the website should not take long to make.

Conclusion

Now you are ready to examine the world with greedy eyes, looking for the absolute simplest ways to figure out what to do!

If you found this post useful, or have examples of when you used simple tests for something, please comment!