Why I'm suss on wellbeing surveys

By Henry Howard🔸 @ 2023-03-18T07:07 (+9)

TL;DR: hunter-gatherers would probably rate themselves at a similar level to us on subjective wellbeing surveys, therefore they're silly.

Big claims from wellbeing surveys

The Happier Lives Institute (HLI) has recently made big claims based on WELLBYs (wellbeing-adjusted life years) interpreted from wellbeing/life-satisfaction surveys.

Their report on StrongMinds (a charity which offers therapy to people in developing countries to improve their mental health) estimates StrongMinds as being between 0.75× to 12×as effective as The Against Malaria Foundation. Great news if true!

Their analyses suggest some surprising findings, such as that the grief from the death of a child equates to a loss of 7.26 WELLBYs [Appendix 2 of The Elephant In The Bednet] while an individual treated in the StrongMinds program can expect a gain of approx. 3 to 20 WELLBYs!!! [Figure 2, Don’t just give well, give WELLBYs: HLI’s 2022 charity recommendation]

HLI have just produced a study supporting the use of wellbeing surveys in charity evaluation: Can we trust wellbeing surveys? A pilot study of comparability, linearity, and neutrality.

 

Something doesn't sit right with me about the use of subjective wellbeing surveys for serious charity analysis. I will try to articulate it with this reductio ad absurdum argument: 

Are you happier than the Sentinelese?

Consider the Sentinelese people of the North Sentinel islands, who are one of the few remaining uncontacted (mostly) peoples in the world, and who probably live with a similar level of technology and culture as the average human did 20,000 years ago.

I ask myself: If asked to rate their life satisfaction in a survey, would the Sentinelese people rate their life satisfaction as significantly lower than mine?

If NO (my personal suspicion):

If YES:

My conclusion is that wellbeing surveys shouldn't be used to compare wellbeing on a global scale, because doing so leads to absurd conclusions. Instead, objective measures of health, wealth, education, and access to amenities should be used at this scale, with wellbeing surveys possibly useful when comparing similar groups.


Samuel Dupret @ 2023-03-20T11:09 (+17)

Hello Henry,

Thank you for presenting this thought experiment. 

The core here is about whether groups like the Sentinelese who do not have the same levels of development as others would give similar levels of SWB. I think the other comments here have done a great job at pointing out possible explanations.

 

Some briefs answers / pointers. Many of these things have been discussed in more details elsewhere.

Charlie_Guthmann @ 2023-03-18T17:53 (+12)

I’m confused why you and everyone else in this thread are so quick to dismiss the idea that hunter gathers have more happiness/ life satisfaction/well being.

This is not at all obvious to me.

alexherwix @ 2023-03-18T20:53 (+10)

I think that it may be helpful to unpack the nature of perceived happiness and wellbeing a little bit more than this post does. I think the idea of hedonic adaptation is pretty well known—most of us have probably heard of the hedonic treadmill (see Brickmann & Campbell, 1971). The work on hedonic adaptation points to the fact that perceived happiness and wellbeing are relative constructs that largely depend on reference points which are invoked. To oversimplify things a little bit, if everyone around me is bad off, I may already be happy if I am only slightly better of than them. At the same time, I might be unhappy if I am pretty good off but everyone around me is much better off. As such, it is entirely reasonable to expect that hunter-gatherers  when asked about their life feel quite good and happy about it as long as they don't feel like everyone else around them is much better off. 

The conclusion of this post should not be that perceived happiness and wellbeing should not be used to compare the effects of interventions but that they simply measure something different than "objective measures". They aim to measure how people feel about their life in general as they compare it to others, not how they score on a particular metric in isolation. Whether you prefer one or the other approach largely depends on your perspective on what is valuable in life. Some people may find making progress on metrics that they find particularly valuable is the way to go and others prefer a more self-organizing perspective where the affected people themselves are more involved in determining what is valuable.

In sum, this post seems a little bit confused on what the WELLBY debate is about. I can recommend the cited article to get some idea on why something like a WELLBY approach may be interesting to consider even if one doesn't like it at first glance.

Brickman, P., & Campbell, D. (1971). Hedonic relativism and planning the good society. In M. H. Appley (Ed.), Adaptation-level theory: A symposium (pp. 287–305). Academic Press. https://archive.org/details/adaptationlevelt0000unse_x7d9/page/287/mode/2up 

alexherwix @ 2023-03-20T09:22 (+2)

Just a short follow up: I just wrote a post on the hedonic treadmill and suggest that it is an interesting concept to reflect about in relation to life in general: 

https://forum.effectivealtruism.org/posts/WMaeBDPdSJLKDzk2d/the-hedonic-treadmill-dilemma-reflecting-on-the-stories-of-1 

Henry Howard @ 2023-03-20T21:20 (+1)

The way you describe WELLBYs - as being heavily influenced by the hedonic treadmill and so potentially unable to distinguish between the wellbeing of the Sentinelese and the modern Londoner - seems to highlight their problems. There's a good chance a WELLBY analysis would have argued against the agricultural revolution, which doesn't seem like a useful opinion.

alexherwix @ 2023-03-21T18:19 (+13)

To me it seems like you have a wrong premise. A wellbeing focused perspective is explicitly highlighting the fact that Sentinelese and the modern Londoners may have similar levels of wellbeing. That's the point! This perspective aims to get you thinking about what is really valuable in life and what the grounds for your own beliefs about what is important are. 

You seem to have a very strong opinion that something like technological progress is intrinsically valuable. Living in a more technically advanced society is "inherently better" and, thus, everyone who does not see this is "objectively wrong". That argument would seem strange to even the most orthodox utilitarian.  Even if your argument is a little bit more nuanced in the sense that you are seeing technological progress only as instrumentally valuable to sustain larger population sizes at similar levels of wellbeing, this perspective is still somewhat naive because technological progress also has potentially devastating consequences such as climate change or AI risks. In that sense, one can actually make the case that the agricultural revolution was maybe the beginning of the end of the human race. So maybe if there would have been a way to grow our societies more deliberately and to optimize for wellbeing (rather than economic growth) from the beginning, it wouldn't have been such a bad idea? I just want to illustrate that the whole situation is not as clear cut as you make it out to be.

Altogether, I would encourage you to keep more of an open mind regarding other perspectives. The post but also this comment of yours make it seem like you might be very quick in dismissing perspectives and being vocal about it even if you have not really engaged with them deeply. This makes you come across as naive to a somewhat more knowledgeable person which could put you at a personal disadvantage in the future and, in addition, could also be contributing to bad epistemics in your community if the people you are talking to are less informed and, thus, not able to spot where you might be cutting corners. Hope you don't resent me for this personal side note, it's meant in a constructive spirit.

NickLaing @ 2023-03-18T09:16 (+10)

First I don't agree with your assumption that hunter gatherers might are likely their wellbeing the same as ours now. The best proxy we might have for "hunter gatherers" today is poorer, less developed countries. People in those  countries have on average have lower average wellbeing than richer countries. My assumption would be in the other direction, that hunter gatherers would most likely rate their wellbeing lower than we would today. 

I don't really understand your argument in this paragraph "Is the difference in WELLBYs significant enough to justify the hundreds of trillions of dollars and hours of effort and suffering (and negative WELLBYs) that have gone (and continue to go) into technological, economic and cultural development to give us our modern lives? 

The answer surely is a resounding yes! If the hunter gatherers rated their wellbeing lower than us and our wellbeing has improved, then surely all that effort into "technological economic and cultural development" is completely worth it!

Vasco Grilo @ 2023-03-18T07:44 (+7)

Hi Henry,

A few notes:

Nathan Young @ 2023-03-18T15:47 (+6)

I think the missing step here is that revealed preference suggest that most Sentinelese people would change their lives to become like us, but we wouldn't change to become like them. So the model doesn't predict what people actually want.

Samuel Dupret @ 2023-03-20T10:24 (+2)

To caveat this:
If we have higher wellbeing, and they are trying to maximise wellbeing, then all these things align.
But people might not be maximising wellbeing or might not be good at it. 

Henry Howard @ 2023-03-18T23:00 (+1)

I agree that revealed preference and survey responses can differ. Unless WELLBYs take account of revealed preferences they'll fail to predict what people actually want

Charlie_Guthmann @ 2023-03-18T17:54 (+1)

Are you claiming that this would be a neccasary thing to prove or this is true but op didn’t include it. I would certainly consider becoming a hunter gatherer if I didn’t have an established life and friends and family