When To Find More Information: A Short Explanation
By Davidmanheim @ 2019-12-28T18:00 (+76)
What do you need to know to make a decision, and when should you try to find out more?
I think this is a really important topic, but when you're making a decision, it's one that deserves very little attention - it's just rather important that it gets that bit of attention.
I wrote my dissertation on the topic, and would argue that basically all questions that matter should be approached using Value-of-Information (VoI). And there are complex quantitative tools that are useful for decisions about this. In doing my dissertation, I tried to actually use some of these methods. I came to the conclusion that you will all live far happier lives if you never actually use them, or even read a dissertation about them. Instead, there are some simple approaches and heuristics that get you almost the whole way there, so I'm writing this post.
Hopefully, this post can help get most of the value at a very low cost. But it will help even more if you use it as an exercise for a specific decision you are making. If possible, pick some personal decision you need to make now - even if it doesn't help, it will be useful practice.
Ready?
VoI for Smart People
You have a decision to make. The fact that you are trying to make a decision implies that the answer isn't obvious - you're unsure about something. Given that, the first step is to spend 5 seconds thinking about whether getting more information would help you make the decision. At this point, you almost always should be able to think of something that you would want to know. (If the answer is that you don't need any, you probably want to think again - or read the above linked post.)
If you think of information you can get easily and at very low cost, get it. Afterwards, if the decision isn't made, come back to the post and start over - is there anything more?
If you discover there are uncertainties you'd like to understand, but it's unclear what you need to know, where to find out more, or how hard it will be to get the information, you should now spend another five minutes, by the clock, really thinking about what you might want to know. But before you start, there are a few ideas to consider.
For that five minutes of thinking, you need to think about what you are trying to decide and how information could help. Here's a set of questions to ask yourself during those five minutes:
1) What decision(s) are you making?
2) What do you not know, specifically, which is important for the decision?
3) If you knew more, would you change your decision? (If not, it's not useful for the decision.)
4) Is there a way to become less uncertain? What could you find out that might help?
5) Is the information that would change your mind worth the cost of gathering it? (This might be tricky, but see below.)
For the last question, usually the answer is obviously yes or no. Sometimes, however, it's unclear, and you need to think a bit more quantitatively about the value of the information. If you want to see the math for how VoI is used in practice, here are some examples, and some more, of how to do the basic quantitative work.
Sometimes, however, you will realize that you need to spend time thinking about the questions and consulting with others before you can start putting numbers together. That's a bit more complex, and if you want to read about how I recommend doing this in great depth, feel free to read the last chapter of the dissertation.
(1) Justifying this would require a much more in-depth discussion than this post warrants, so feel free to ignore the claim.
(2) Specifically Bayesian Markov Chain Monte Carlo simulation, and Discrete Bayesian Networks. It's all in the dissertation - which you shouldn't read (see below.)
(3) If you want to put yourself to sleep, feel free to read anything other than the first 5 pages of the fifth chapter of the dissertation. (4)
(4) Really, only pages 159-164, here. That little bit will provide 90% of the value. That's the part where I point out that doing the basics is a better use of your time than quantitative modeling. Trust me - the rest of the information in the dissertation is low-value, and for almost everyone, it's not worth the cost of reading.
Khorton @ 2019-12-28T20:19 (+16)
This was a really great Forum post! You expressed yourself really clearly, and I understood why you didn't provide further information and what I should do next. Thanks so much!
Benjamin_Todd @ 2019-12-30T20:20 (+11)
5) Is the information that would change your mind worth the cost of gathering it? (This might be tricky, but see below.)
For the last question, usually the answer is obviously yes or no. Sometimes, however, it's unclear, and you need to think a bit more quantitatively about the value of the information. If you want to see the math for how VoI is used in practice, here are some examples, and some more, of how to do the basic quantitative work.
Thanks for the post, but this seems like the tricky bit to me. Might you be able to give some rough rules of thumb people could apply to answer this question?
Trying to do actual VOI estimates gets pretty confusing, so what would be great is something simpler than that, but better than just going with your intuition.
I think there are probably some things to say, along the lines of things like "if you can spend under 10% of the time at stake in the decision, and you think it's likely you'd change your mind (say with 50% chance), then probably investigate more"; or "if you're early in your career, leans towards investigating because information is more valuable to you"; "people typically consider too few options, so it's worth generating at least one alternative to your current options".
Davidmanheim @ 2019-12-31T10:38 (+13)
This is an area I should probably write more about, but I have a harder time being pithy, and haven't tried to distill my thoughts enough. But since you asked....
As a first approximation, you want to first consider the plausible value of the decision. If it's choosing a career, for example, the difference between a good choice and a bad one is plausibly a couple million dollars. You almost certainly don't want to spend more than a small fraction of that gathering information, but you do want to spend up to, say, 5% on thinking about the decision. (Yes, I'd say spending a year or two exploring the options before picking a career is worthwhile, if you're really uncertain - but you shouldn't need to be. See below.)
Once you have some idea of what the options are, you should pick what about the different options are good or bad - or uncertain. This should form the basis of at least a pro/con list - which is often enough by itself. (See my simulation here.) If you see that one option is winning on that type of list, you should probably just pick it - unless there are uncertainties that would change your mind.
Next, list those key uncertainties. In the career example, these might include: Will I enjoy doing the work?, How likely am I to be successful in the area?, How likely is the field to continue to be viable in the coming decades?, and How easy or hard is it to transition into/out of?
Notice that some of the uncertainties matter significantly, and others don't. We have a tool that's useful for this, which is the theoretical maximum of VoI, called Value of Perfect Information. This is the difference in value between knowing the answer with certainty, and the current decision. (Note: not knowing the future with certainty, but rather knowing the correct answer to the uncertainty. For example, knowing that you would have a 70% chance of being successful and making tons of money in finance.) Now ask yourself: If I knew the answer, would it change my decision? If the answer is no, drop it from the list of key uncertainties. If a relatively small probability of success would still leave finance as your top option, because of career capital and the potentially huge payoff, maybe this doesn't matter. Alternatively, if even a 95% chance of success wouldn't matter because you don't know if you'd enjoy it, it still doesn't matter - so move on to other questions.
If the answer is that knowing the answer would change your mind, you need to ask what information you could plausibly get about the question, and how expensive it is. For instance, you currently think there's a 50% chance you'd enjoy working in finance. Spending a summer interning would make you sure one way or the other - but the cost in time is very high. It might be worth it, but there are other possibilities. Spending 15 minutes talking to someone in the field won't make you certain, but will likely change your mind to think the odds are 90% or 10% - and in the former case, you can still decide to apply for a summer internship, and in the latter case, you can drop the idea now.
You should continue with this process of finding key things that would change your mind until you either think that you're unlikely to change your mind further, or the cost of more certainty is high enough compared to the value of the decision that it's not obviously worth the investment of time and money. (If it's marginal or unclear, unless the decision is worth tens or hundreds of millions of dollars, significant further analysis is costly enough that you may not want to do it. If you're unsure which way to decide at that point, you should flip a coin about the eventual decision - and if you're uncertain enough to use a coin flip, then just do the riskier thing.)
Benjamin_Todd @ 2019-12-31T15:19 (+4)
Thanks - it's useful to see your take on this!
Milan_Griffes @ 2019-12-30T20:43 (+2)
Trying to do actual VOI estimates gets pretty confusing
+1
From my (limited) experience with VOI, it sounds great in theory, but in practice quickly turns into a hopeless pedantic muddle.
Davidmanheim @ 2019-12-31T10:10 (+1)
Yes, that was partially the conclusion of my dissertation - and see my response to the above comment.
Davidmanheim @ 2021-12-19T18:00 (+6)
I think that it's generally useful to share a clear paradigm which is useful for non-experts, based on deep knowledge of a subject, and that's what I tried to do here. In this case, I think that the concept and approach are a very generally useful point, and I would be very excited for more people to think about Value of Information more often, though as the post notes, mostly informally.
cole_haus @ 2020-01-03T02:24 (+2)
I may be falling prey to the curse of knowledge since I already knew about VoI when I read it but I thought How to Measure Anything (which has a chapter on VoI) was very pragmatic and accessible.
If you want to see the math for how VoI is used in practice, here are some examples, and some more, of how to do the basic quantitative work.
Also, not sure if it's actually useful or falls into the "complex quantitative tools ... [that you shouldn't use]" category but I have a basic VoI calculator at: https://www.col-ex.org/posts/value-of-information-calculator-explained/.
Davidmanheim @ 2020-01-04T18:34 (+4)
Good writeup, and cool tool. I may use it and/or point to it in the future.
I agree that when everything is already quantified, and you can do this. The chapter in HtMA is also fantastic. But it's fairly rare that people have already quantified all of the relevant variables and properly explored what the available decisions are or what they would affect - and not doing so can materially change the VoI, and are far more important to do anyways.
That said, no, basic VoI isn't hard. It's just that the use case is fairly narrow, and the conceptual approach is incredibly useful in the remainder of cases, even those cases where actually quantifying everything or doing to math is incredibly complex or even infeasible.