DISCLAIMER: (perhaps a double edge sword) I've lived in Uganda here for 10 years working  in Healthcare.

Thanks Michael for all your efforts. I love StrongMinds and am considering donating myself.  I run health centers here in Northern Uganda and have thought about getting in touch with you see if we can use something like the Strong minds in the health centers we manage.  While working as a doctor here my estimate from experience that for perhaps between 1 in 5 and 1 and 10 of our patients, depression or anxiety is the biggest medical problem in their lives.  I feel bad every year that we do nothing at all to help these people.

Point 1

First I read a reply below that seriously doubted that improving depression could have more  positive psychological effect than preventing the grief of the death of a child.  On this front I think it's very hard to make a call in either direction, but it seems plausible to me that lifting someone out of depression could have a greater effect in many cases.

Point 2

I however strongly disagree with your statement here about self reporting. Sadly I think it is not a good  measure especially as a primary outcome measure.

"Also, what's wrong with the self-reports? People are self-reporting how they feel. How else should we determine how people feel? Should we just ignore them and assume that we know best? Also, we're comparing self-reports to other self-reports, so it's unclear what bias we need to worry about."

Self reporting doesn't work because poor people here in Northern Uganda at least are primed to give low marks when reporting how they feel before an intervention, and then high marks afterwards - whether the intervention did anything or not. I have seen it personally here a number of times with fairly useless aid projects. I even asked people one time after a terrible farming training, whether they really thought the training helped as much as they had reported on the piece of paper. A couple of people laughed and said something like "No of course it didn't help, but if we give high grades we might get more and better help in future". this is an intelligent and rational response by recipients of aid, as of course good reports of an intervention increase their chances of getting more stuff in future, useful or not.

 Dambisa Moyo says it even better in her book "Dead Aid", but  couldn't find the quote. There might also be good research papers and other effective altruism posts that describe this failing of self reporting better than me so apologies if this is the case.

You also said "Also, we're comparing self-reports to other self-reports", which doesn't help the matter, because those who don't get help are likely to keep scoring the survey lowly because they feel like they didn't get help

Because of this I struggle to get behind any assessment that relies on self-reporting, especially in low income countries like Uganda where people are often reliant on aid, and desperate for more.  Ironically perhaps I have exactly the same criticism of GiveDirectly. I think  that researchers of GiveDirectly should use exclusively (or almost exclusively) objective measures of improved life (hemoglobin levels, kids school grades, weight for height charts, assets at home) rather the before and after surveys they do.  To their credit, recent GiveDirectly research seem to be using more objective measures in their effectiveness research.

https://www.givedirectly.org/research-at-give-directly/

We can't ignore how people feel, but we need to try and find objective ways of assessing it, especially in contexts like here in Uganda where NGOs have wrecked any chance of self reporting being very accurate. I feel like measuring improvement in physical aspects of depression could be a way forward. Just off the top of my head you could measure before and after mental agility scores, which should improve as depression improves, or quality of sleep before and after using a smart watch or phone. Perhaps even you could use continuous body monitoring for a small number of people, as they did here

https://www.vs.inf.ethz.ch/edu/HS2011/CPS/papers/sung05_measures-depression.pdf
 

Alternatively I'd be VERY interested in a head to head Cash transfer vs Strongminds RCT - should be pretty straightforward , even potentially using your same subjective before and after scores. Surely this would answer some important questions.

A similar comparative RCT was done in Kenya in 2020 of cash transfer vs. Psychotherapy, and the cash transfers clearly came through on top  https://www.nber.org/papers/w28106.

Anyway I think Strong minds is a great idea and probably works well to the point I really want to use it myself in our health centers, but I don't like the way you measure it's effectiveness and therefore doubt whether it is as effective as stated here. 

Thanks for all the good work!

JoelMcGuire @ 2022-11-28T21:10 (+9)

Hi Nick, 

Thank you for sharing! This is valuable to hear. The issue of being primed to respond in a certain way has surprisingly not been explored widely in low-income countries.

We’re concerned about this, but to our knowledge, the existing evidence suggests this isn’t a large concern. The only study we’ve seen that explicitly tries to measure the impact of this was a module in Haushofer et al., (2020 section III.I) – where they find about a zero effect. If there’s more evidence we don’t know about, please share! 

Here's the excerpt (they call these effects "experimenter demand effects"). 

You also said "Also, we're comparing self-reports to other self-reports", which doesn't help the matter, because those who don't get help are likely to keep scoring the survey lowly because they feel like they didn't get help
 

Allow me to disagree -- I think this could help the matter when we're comparing between interventions. If this effect was substantial,  the big concern would be if this effect differs dramatically between interventions. E.g., the priming effect is larger for psychotherapy than for cash transfers. I.e., it's okay if the same bias is balanced across interventions. Of course, I'm not sure if it's plausible for this to be the case -- my prior is pretty uniform here. 

All that being said, I totally endorse more research on this topic!

We can't ignore how people feel, but we need to try and find objective ways of assessing it, especially in contexts like here in Uganda where NGOs have wrecked any chance of self reporting being very accurate.

I think this is probably a point where we disagree. A point we've expanded upon previously (see McGuire et al., 2022) is that we seriously doubt that we can measure feelings objectively. The only way we can know if that objective measure is capturing a feeling is by asking someone if it does so -- so why not just do that?  

 I am much more pro "find the flaws and fix them" then "abandon ship when it leaks" regarding measuring subjective wellbeing. 

 

Alternatively I'd be VERY interested in a head to head Cash transfer vs Strongminds RCT - should be pretty straightforward , even potentially using your same subjective before and after scores. Surely this would answer some important questions.
 

This may be in the works! 

NickLaing @ 2022-11-29T10:53 (+13)

Thanks so much Joel and I’m stoked by your response. I don’t think I’ve been in a forum where discussion and analysis is this good. 

I'm not sure that having close to no evidence on a major concern about potential validity of before and after surveys should be very reassuring.

That tiny piece of evidence you cited only looks at the “experimenter demand effect”, which is a concern yes, but not my biggest concern. My largest concern is let's say the “future hope” effect which I gave the example from in my first reply – where participants rate the effect of interventions more positively than their actual experience, because they CORRECTLY assess that a higher rating may bring them  better help in future. That’s what I think is most likely to wreck these before and after surveys. 

I don’t know this field well at all like you so yes it seems likely this is a poorly researched issue . We have experience and anecdotes like those from Dambisa Moyo, me and many others working in the NGO world, that these incentives and vested interests can greatly affect before and after surveys. You recently wrote an article which included (to your credit) 4 disadvantages of the WELLBY. My inclination is (with enormous uncertainty) that these  problems I've outlined with before and after subjective surveys in low income countries are at least as big a disadvantage to the WELLBY approach as any of the 4 issues you outlined. I agree that SWB is a well validated tool for one time surveys and there are no serious vested interests involved. It’s the before and after surveys in low income countries that are problematic.

These are the 3 major un-accounted for problems I see with before and after self reporting (first 2 already discussed).
 

1. My biggest concern is the “future hope” effect - people positively responding because they correctly believe a positive response is likely to get them more and even better help in future. 
2. The “ experimenter demand effect” (as you discussed) is if interviewees are primed to give people the answer they think the experimenters want. 
    
3. A third potential problem (which I haven’t mentioned already but have experienced) is that interviewers manipulate results positively in the direction of the intervention either through the way they ask questions, or even through increasing scores fraudulently. This is again rational as their jobs often depend on the NGO raising more money to support “successful interventions” to keep them employed. I have seen this here a number of times during surveys within my own organisation AND in other organisations, It’s very hard to stop this effect even after talking to researchers about it – I have tried but there was still manipulation present. This is probably a smaller problem than the other two, and easier to control and stamp out.
    

On the "Also, we're comparing self-reports to other self-reports" front - I agree this could be be fine if comparing between two interventions (e.g. cash and psychotherapy) and the effect might be similar between interventions. I think though most of the studies that you have listed  compare intervention to no intervention, so your point may not stand in these cases.

I’ll change my mind somewhat in your direction and give you some benefit of the doubt on your point about objective measures not working well for wellbeing assessment, given that I haven’t researched it very well and I'm not an expert. Let's leave objective measures out of the discussion for the moment!

I love the idea of your RCT cash transfers vs. psychotherapy but I'm confused about a number of parts of the design and have a few questions if you will humour me .

- The study design seems to say you are giving cash only the intervention groups and not the control group? I suspect this is a mistake in the reporting of the study design but please clarify. To compare cash vs psychotherapy would you not give the cash to the whole control group and either not to the intervention group at all, or only a small subsection of the intervention group? I might well have missed something here...
- Why are you giving money at the end of the intervention rather than at the start? Does  that not give less time for the benefits of the cash to take effect.
- Why are you only giving $50 cash transfer and not around $130 (the cost of the therapy). Would it not would be a fairer comparison to compare like for like in terms of money spent on the intervention?

It seems logical that most RCTs in the effective altruism space now should be intervention vs cash transfer, or at least having 3 arms Intervention vs cash transfer vs nothing. Hopefully I've read it wrong and that is the case in this study!

To finish I like positivity and  I'll get alongside your point about fixing the boat rather than dismissing it. I feel like the boat is more than a little leak at this stage, but I hope I'm wrong I love the idea of using before and after subjective wellness measures to assess effectiveness of interventions, I'm just not yet convinced yet it can give very meaningful results based on my issues above. 

Thanks so much if you got this far and sorry it's so long!

JoelMcGuire @ 2022-11-29T18:08 (+5)

A quick response -- I'll respond in more depth later, 

I love the idea of your RCT cash transfers vs. psychotherapy but I'm confused about a number of parts of the design and have a few questions if you will humour me .
 

To be clear, the planned RCT has nothing to do with HLI, so I would forward all questions to the authors! :)

And to be clear when you say "before and after measures", do you think this applies to RCTs where the measurement of an effect is comparing a control to a treatment group instead of before and after a group receives an intervention? 

NickLaing @ 2022-11-29T19:04 (+5)

Apologies for assuming that the RCT involved HLI - the strong minds involvement lead me to that wrong assumption! They will then be addressing different questions than we are here - unfortunately for us that trial as stated in the protocol doesn't test cash transfers vs. psychotherapy.

I don't quite understand the distinction in your question sorry, will need rephrasing! I'm referencing  the problems with any SWB measurement which involves measuring SWB at baseline and then  after an intervention. Whether there is a control arm or not.

Looking forward to hearing more nice one!

JoelMcGuire @ 2022-12-02T01:29 (+4)

Right, our concern is that if this bias exists, it is stronger for one intervention than another. E.g., say  psychotherapy is more susceptible than cash transfers. If the bias is balanced across both interventions, then again, not as much of an issue.  

I'm wondering if this bias your concerned with would be captured by a "placebo" arm of an RCT. Imagine a control group that receives an intervention that we think has no to little effect. If you expect any intervention to activate this "future hope" bias, then we could potentially estimate the extent of this bias with more trials including three arms: a placebo,  a receive nothing, and an intervention arm. 

Do you have any ideas on how to elicit this bias experimentally? Could we instruct interviewers to, for a subsample of the people in a trial, explicit something like "any future assistance [ will / will not] depend on the benefit this intervention provided." Anything verbal like this would be cheapest to test. 

NickLaing @ 2022-12-02T08:53 (+6)

Hi Joel - Nice one again

"Right, our concern is that if this bias exists, it is stronger for one intervention than another. E.g., say  psychotherapy is more susceptible than cash transfers. If the bias is balanced across both interventions, then again, not as much of an issue."

I would have thought the major concern would have been if the bias existed at all, rather than whether it balanced between interventions. Both StrongMinds evidence assessing their own program and most of the studies used in your WELLBY analysis are NOT vs. another intervention, but rather Psychotherapy vs. No intervention. This is often labelled "Normal care", which is usually nothing in low income countries.  So if it exists at all in any magnitude, it will be affecting your results.

Onto your question though, which is still important but I believe of secondary importance

Your idea of a kind of "fake" placebo arm would work - providing the fake placebo was hyped up as much and taken as seriously as the treatment arm, AND that the study participants really didn't know it was a placebo. Unfortunately you can't do this as It's not ethical to have an RCT with an intervention that you think has no or little effect. So not possible I don't think

I like your idea of  interviewers in a trial stating for a subset of people that their answers won't change whether they get more or not n future. I doubt this would mitgate the effect much or at all, but it's a good idea to try!

My very strong instinct is that cash transfers would illicit a FAR stronger effect than psychotherapy. It's hard to imagine anything that would illicit "future hope" more than the possibility of getting cash in future. This seems almost self-evident. Which extremely poor person in their right mind (whether their mental health really is better or not) is going to say that cash didn't help their mental health if they think it might increase their chance of getting more cash in future?

Again I think just direct RCTs vs. Cash Transfers is the best way to test your intervention and control for this bias. It's  hard to imagine anything having a larger "future hope"  effect than cash. If psychotherapy really beat the cash given that cash will almost certainly have a bigger "future hope" bias than the psychotherapy, then I'd say you'd have a slam dunk case. 

I am a bit bewildered that StrongMinds has never done a psychotherapy vs. Cash transfer trial. This is the big question, and you claim that Strongminds therapy produces more WELLBYs than Cash Transfers yet there is no RCT?  It looks like that trial you sent me is another painful missed opportunity to do that research as well. Why there is no straight cash transfer arm in that trial doesn't make any sense.

As far as I can see the Kenyan trial is the only RCT with Subjective wellness assessment which pits Cash Transfers vs. Psychotherapy (although it was a disproportionately large cash transfer and not the Strongminds style group therapy). If group Psychotherapy beat an equivalent cash Cash transfer in say 2 RCTs I might give up my job and start running group psychotherapy sessions - that would be powerful!

BarryGrimes @ 2023-01-02T20:25 (+13)

Hi Nick. I found more details about the Baird et al. RCT here. I've copied the section about the 'cash alone' arm below as I know you'll be interested to read that:

One of the issues that can cause difficulties in the interpretation of the findings from this study comes from the fact that it does not have a classical 2x2 factorial design that includes a “cash alone” arm. While it would have been ideal, from a study design perspective, to have such a design, which would have enabled us to experimentally reject (or not) that cash alone would have been as effective as IPT-G+. Such benchmarking of the (cost-) effectiveness of a group therapy intervention to cash transfers alone would have been nice, the three-arm trial we designed for our setting, leaving out a “cash-only” arm, is an adequate study design for the following three reasons.
 

First, the extant evidence on the effects of economic interventions in general, or cash transfers in particular, do not support the idea of improved mental health outcomes past the short-run. For example, in their review, Lund et al. (2011) finds that the mental health effects of poverty alleviation programs were inconclusive. Blattman, Jamison, and Sheridan (2017) find no effects of lump-sum cash transfers alone on mental health outcomes in the short- or the medium-run. Short-term effects of cash transfers (monthly or lump-sum) on psychological wellbeing that were observed in the short-run dissipated a couple of years later (Baird, de Hoop, and Özler 2013; Baird, McIntosh and Özler 2019; Haushofer and Shapiro 2016, 2018). Hence, we do not think that there is sufficient equipoise to include a “cash only” arm when it comes to sustained effects on depression two-years after the end of the intervention.
 

Second, even if one could make a case that there may be a sustained income effect on mental health – for example, such as those indicated by studies of lottery winners (Gardner and Oswald 2007; Lindahl 2005) – the amounts offered to IPT-G participants here are too small to have this kind of an effect two years after they are transferred. A similar argument has been made in another experiment that lacked a pure unconditional cash transfer (UCT) arm: Benhassine et al. (2015) argue that the labelling of the cash transfer as an education support program increased school participation through its effect on the “...parents’ belief that education was a worthwhile investment,” rather than through a pure income effect, because the transfers were too small to cause the observed effects.
 

Finally, and related to the point above, one of the aims of our trial is to test the efficacy of a low-cost and scalable intervention through two NGOs that have a track record of implementing programs that are being utilized here (BRAC Uganda and StrongMinds Uganda). While BRAC Uganda is interested in taking advantage of its ELA girls’ clubs’ platform to provide mental health services across Uganda, it does not have any plans to provide UCTs, especially not in transfer sizes that might perhaps have sustained effects. Hence, the lack of evidence on the potential effectiveness of UCTs on sustained reductions in depression, combined with a lack of interest from the implementing partners, resulted in the study team designing a trial that has only three arms. Should the trial show that IPT-G+ is significantly more effective than IPT-G alone in reducing depression in the medium-run, our interpretation will be that there is a complementarity between the two interventions, and not that cash is effective on its own for sustained improvements in psychological wellbeing.

NickLaing @ 2023-01-03T06:48 (+5)

Thanks Barry I tried to find this earlier but couldn't.

I find these arguments rather uncompelling. What do you think Barry and Joel? (I wish I could tag people on this forum haha)

That they feel the need to write 4 paragraphs to defend against this elephant in the room says a lot. The question we are all still asking is how much better (if at all) StrongMinds really is than cash for wellbeing. 

My first question is why don't they reference the 2020 Haushofer study, the only RCT comparing psychotherapy to cash and showing cash is better?  https://www.nber.org/papers/w28106

Second, their equipoise argument is very poor. The control arm should have been BRAC ELA club + cash.  Then you keep 3 arms and avoid their straw man 4 arm problem. You would lose nothing  in equipoise giving cash to the control arm - I don't understand the equipoise argument perhaps I'm missing something?

Then third there's this...

"Should the trial show that IPT-G+ is significantly more effective than IPT-G alone in reducing depression in the medium-run, our interpretation will be that there is a complementarity between the two interventions, and not that cash is effective on its own for sustained improvements in psychological wellbeing."

This is the most telling paragraph. It's like, we designed our study so that even if we see that cash gives a big boost, we aren't going to consider the alternative that we don't like. It seems to me like they are defending poor design post-hoc, rather than that they made good decision made in advance. 

The more I see this, the more I suspect  that leaving the cash arm out was either a big mistake or an intentional move by the NGOs. What we have now is a million dollar RCT, which doesn't answer conclusively the most important question we are all asking. This leaves organisations like your HLI having to use substandard data to assess psychotherapy vs. cash because there is no direct  gold standard comparison.

It's pretty sad that a million dollars will be spent on a study that at best fails to address the elephant in the room (while spending 4 paragraphs explaining why they are not).  Other than that the design and reasoning in this study seems fantastic. 

Matt_Lerner @ 2022-11-28T21:13 (+5)

Hey Nick, thanks for this very valuable  experience-informed comment. I'm curious what you make of the original 2002 RCT that first tested IPT-G  in Uganda. When we (at Founders Pledge) looked at StrongMinds (which we currently recommend, in large part on the back of HLI's research), I was surprised to see that the results from the original RCT lined up closely with the pre/post scores reported by recent program participants.

Would your take on this result be that participants in the treated group were still basically giving  what they saw as socially desirable answers, irrespective of the efficacy of the intervention? It's true that the control arm in the 2002 RCT did not receive a comparable placebo treatment, so that does seem a reasonable criticism. But if the socially desirability bias is so strong as to account for the massive effect size reported in the 2002 paper, I'd expect it to appear in the  NBER paper you cite, which also featured a pure control group. But that paper seems to find no effect of psychotherapy alone.

NickLaing @ 2022-11-29T11:23 (+2)

Matt these are fantastic questions that I definitely don't have great answers to, but here are a few thoughts.

First I'm not saying at all that the Strong minds intervention is likely useless - I think it is likely very useful. Just that the positive effects may well be grossly overstated for the reasons outlined above.

My take on the result of that original 2002 RCT and Strong Minds. Yes like you say in both cases it could well be that the treatment group are giving positive answers both to appease the interviewer (Incredibly the before and after interviews were done by the same researcher in that study which is deeply problematic!)  and because they may have been hoping positive responses might provide them with further future help.

Also in most of these studies, participants are given something physical for being part of the intervention groups. Perhaps small allowances for completing interviews, or tea and biscuits during the sessions. These tiny physical incentives can be more appreciated than the actual intervention. Knowing World  Vision this would almost certainly be the case

I have an immense mistrust of World vision for a whole range of reasons, who were heavily involved in that famous 2002 RCT. This is due to their misleading advertising and a number of shocking experiences of their work here in Northern Uganda which I won't expand on. I even wrote a blog about this a few years ago, encouraging people not to give them money.  I know this may be a poor reason to mistrust a study but my previous experience heavily biases me all the same.

Great point about the NBER paper which featured a pure control group. First it was a different intervention - individual CBT not group therapy. 

Second it feels like the Kenyan study was more dispassionate than some of the other big ones. I might be wrong but a bunch of the other RCTs are partly led and operated by organisations with something to prove. I did like that the Kenyan RCT felt less likely to be biased as there didn't seem to be as much of an agenda as with some other studies.

Third, the Kenyan study didn't pre-select people with depression,  the intervention was performed on people randomly selected from the population. Obviously this means you are comparing different situations when comparing this to the studies with group psychotherapy for people with depression.

Finally allow me to speculate with enormous uncertainty. I suspect having the huge 1000 dollar cash transfers involved really changed the game here. ALL participants would have known for sure that some people people were getting the cash and this would have changed dynamics a lot. One outcome could have been that other people getting a wad of cash might  have devalued the psychotherapy in participants eyes. Smart participants may even have decided the were more likely to get cash in future if they played down the effect of the therapy. Or even more extreme the confounding could go in the opposite direction of other studies, if participants assigned to psychotherapy undervalued a potentially positive intervention, out of disappointment at not getting the cash in hand. Again really just summising, but never underrate the connectivity and intelligence of people in villages in ths region!

Thanks for the post Michael — these sorts of posts have been very helpful for making me a more informed donor. I just want to point out one minor thing though.

I appreciate you and your team's work and plan on donating part of my giving season donations to either your organisation, Strongminds, or a combination of both. But I did find the title of this post a bit unnecessarily adversarial to GiveWell (although it's clever, I must admit).

I've admired the fruitful, polite, and productive interactions between GW and HLI in the past and therefore I somewhat dislike the tone struck here.

TL;DR: This post didn't address my concerns related to using WELLBYs as the primary measurement of how much an intervention increases subjective wellbeing in the short term, so in this comment I explain my reasons for being skeptical of using WELLBYs as the primary way to measure how much an intervention increases actual wellbeing.

~~~~

In Chapter 9 ("Will the Future Be Good or Bad?") of What We Owe the Future, Will MacAskill briefly discusses life satisfaction surveys and raises some points that make me very skeptical of HLI's approach of using WELLBYs to evaluate charity cost-effectiveness, even from a very-short-term-ist hedonic utilitarian perspective.

Here's an excerpt from page 196 of WWOTF, with emphasis added by me:

We can't assume that [the neutral wellbeing point] is the midpoint of the scale. Indeed, it's clear that respondents aren't interpreting the question literally. The best possible life (a 10) for me would be one of constant perfect bliss; the worst possible life (a 0) for me would be one of the most excruciating torture. Compared to these two extremes, perhaps my life, and the lives of everyone today, might vary between 4.9 and 5.1. [William Kiely note: It seems like the values should vary between 1.4-1.6, or around whatever the neutral point is, not around 5, which MacAskill is about to say is not the neutral point.] But, when asked, people tend to spread their scores across the whole range, often giving 10s or 0s. This suggests that people are relativising their answers to what is realistically attainable in their country or the world at present. A study from 2016 found that respondents who gave themselves a 10 out of 10 would often report significant life issues. One 10-out-of-10 respondent mentioned that they had an aortic aneurysm, had had no relationship with their father since his return from prison, had had to take care of their mother until her death, and had been in a horrible marriage for seventeen years.

The relative nature of the scale means that it is difficult to interpret where the neutral point should be, and unfortunately, there have been only two small studies directly addressing this question. Respondents from Ghana and Kenya put the neutral point at 0.6,while one British study places it between 1 and 2. It is difficult to know how other respondents might interpret the neutral point. If we take the UK survey on the neutral point at face value, then between 5 and 10 percent of people in the world have lives that are below neutral. All in all, although they provide by far the most comprehensive data on life satisfaction, life satisfaction surveys mainly provide insights in relative levels of wellbeing across different people, countries, and demographics. They do not provide much guidance on people's absolute level of wellbeing.

Some context on the relative value of different conscious experiences:

Most people I have talked to think that the negative wellbeing experiences they have had tend to be much worse than their positive wellbeing experiences are good.

In addition to thinking this about typical negative experiences compared to typical positive experiences, most people I talk to also seem to think that the worst experience of their life was several times more bad than their best experience was good.

People I talk to seem to disagree significantly on how much better their best experiences are compared to their typical positive experience (again by "better" I mean only taking into account their own wellbeing, i.e. the value of their conscious experience). Some people I have asked say their best day was maybe only about twice as good as their typical (positive) day, others think their best day (or at least best hour or best few minutes) are many times better (e.g. ~10-100 times better) than their typical good day (or other unit of time).

In the Effective Altruism Facebook group 2016 poll "How many days of bliss to compensate for 1 day of lava-drowning?" (also see 2020 version here), we can see that EAs' beliefs about the relative value of the best possible experience and the worst possible experience span many orders of magnitude. (Actually, answers spaned all orders of magnitude, including "no amount of bliss could compensate" and one person saying that even lava-burning is positive value.)

Given the above context, why 0-10 measures can't be taken literally...

It seems to me that 0-10 measures taken from subjective wellbeing / life satisfaction surveys clearly cannot be taken literally.

That is, survey respondents are not answering on a linear scale. An improvement from 4 to 5 is not that same as an improvement from 5 to 6.

Respondents' reports are not comparable to each others'. One person's 3 may be better than another's 7. One person's 6 may be below neutral wellbeing, another person's 2 may be above neutral wellbeing.

The vast majority of respondents' answers presumably are not even self-consistent either. A "5" report one day is not the same as a "5" report a different day, even for the same person.

If the neutral wellbeing point is indeed around 1-2 for most people answering the survey, and peoples' worst experiences are much worse than their best experiences are good (as many people I've talked to have told me), then such surveys clearly fail to capture that improving someone's worst day to a neutral wellbeing today is much better than making someone's 2 day into a 10 day. That is, it's not the case that an improvement from 2 to 10 is five times better than an improvement from 0 to 2 in many cases, as a WELLBY measurement would suggest. In fact, the opposite may be true (with the improvement from 0 to 2 (or whatever neutral wellbeing is) potentially being 5 times greater (or even more times greater) than an improvement from 2 to 10. This is a huge discrepancy and I think gives reason to think that using WELLBY's as the primary tool to evaluate how much interventions increase wellbeing is going to be extremely misleading in many cases.

What I'm hearing from you

I see:

Lots of research has shown that subjective wellbeing surveys are scientifically valid (e.g. OECD, 2013; Kaiser & Oswald, 2022).

As a layperson note that I don't know what this means.

(My guess (if it's helpful to you to know, e.g. to improve your future communications to laypeople) is that the "scientifically valid" means something like "if we run an RTC in which we give a control group a subjective wellbeing survey and another group that we're doing some intervention on to make them happier the same survey, we find that the people who are happier give higher numbers on the survey. Then later when we run this study again, we find consistent results with people giving higher scores in approximately the same range for the same intervention, which we interpret to mean that the self-reported wellbeing is actually a measurement of something real.)

Despite not being sure what it means for the surveys to be scientifically valid, I do know that I'm struggling to think of what it could mean such that it would overcome my concerns above about using subjective wellbeing surveys as the main measure of how much an intervention improves wellbeing.

Peoples' 0-10 subjective wellbeing reports seem like they are somewhat informative about actual subjective wellbeing—e.g. given only information about two people's self-reported subjective wellbeing I'd expect the wellbeing of the person with the higher reported wellbeing to be higher—but there are a host of reasons to think that 1-point increases in self-reported wellbeing don't correspond to actual wellbeing being increased by some consistent amount (e.g. 1 util) and reading this post didn't give me reason to think otherwise.

So I still think a cost-effectiveness analysis that uses subjective wellbeing assessments as more than just one small piece of evidence seems very likely to fail to identify what interventions actually increase subjective wellbeing the most. I'd be interested in reading a post from HLI that advocates for their WELLBYs approach in light of the sort of the concerns mentioned above.

Thanks for this post, fascinating stuff!

My quick-ish question: is it possible that you are underestimating the WELLBY effect of grief, for AMF? My understanding (from referring back to the 'elephant in the bednet' post, but totally possible that I've missed something) is that these estimates are coming from Oswald and Powdthavee (2008), and then assuming a 5 year duration from Clark et al. (2018). Hence getting an estimate of ~ 7 WELLBYs.

The reason I'm a little skeptical of this is first that it seems likely to me (disclaimer that I have not done a deep dive) that losing a child would increase the likelihood of depression and other mental illnesses, alongside other things like marriage disruption (e.g. Rogers et al. 2008, which highlights effects lasting to the 18 year mark). I don't think these effects will be accounted for by pulling out the estimate coming from Oswald & Powdthavee according to the Clark paper.

Along similar lines, I also think the spillover effects of AMF might be underestimated;  my intuition is that losing a child is inherently especially shocking, and that the spillover effects might  larger than the spillover effects from things like cash transfers/ therapy- e.g. everyone in the community feels sad (to some degree) about it.  Am I correct that the spillover for AMF is calculated only for family members, not for friends and other members of the community?

This is great to hear! I'm personally more excited by quality-of-life improvement interventions rather than saving lives so really grateful for this work.

Echoing kokotajlod's question for GiveWell's recommendations, do you have a sense of whether your recommendations change with a very high discount rate (e.g. 10%)? Looking at the graph of GiveDirectly vs StrongMinds it looks like the vast majority of benefits are in the first ~4 years

Minor note: the link at the top of the page is broken (I think the 11/23 in the URL needs to be changed to 11/24)

I feel very uncomfortable mutiplying, dividing adding up these wellbys as if they are interchangeable numbers.

I have skimmed this - but if I'm reading the graph right... you believe therapy generates something like 3 wellbys of benefit per attendee?

And you say in your elephant bednet report "We estimate the grief-averting effect of preventing a death is 7 WELLBYs for each death prevented"

So a person attending therapy is roughly equivalent to preventing one parent's grief at death of a child?

This doesn't seem plausible to me.

Apologies if I have misunderstood somewhere.

MichaelPlant @ 2022-11-25T08:58 (+5)

I feel very uncomfortable mutiplying, dividing adding up these wellbys as if they are interchangeable numbers.

Yeah, it is a bit weird, but you get used to it! No weirder than using QALYs, DALYs etc. which people have been doing for 30+ years. 

Re grief, here's what we say in section A.2 of the other report

We do a shallow calculation for grief in the same way we did in Donaldson et al. (2020). The best estimate we found is from Oswald and Powdthavee (2008): a panel study in the UK which finds the effect on life satisfaction due to the death of a child in the last year as being -0.72 (adjusted for a 0-10 scale). According to Clark et al. (2018), the duration of grief is ~5 years. Based on data from the UNDP, we calculate that the average household size across the beneficiary countries (excluding the recipient of the nets) is 4.03 people (row 16). Hence, an overall effect of grief per death prevented is (0.72 x 5 x 0.5) x 4.03 = 7.26 WELLBYs. However, we think this is an upper bound because it doesn’t account for the counterfactual grief averted. If you avert the death of someone, they will still die at some point in the future, and the people who love them will still grieve.

I'm not sure what you find weird about those numbers.  

Making this account feels almost as bad as pulling a "Holden", @ 2022-11-26T01:29 (+3)

As described elsewhere, the approach of measuring happiness/sentiment in a cardinal way and comparing this to welfare from pivotal/tragic life events measured in years/disability, seems challenging and the parent comment's concerns about the magnitudes seems dubious is justified. 

GHD is basically built on a 70-year graveyard of very smart people essentially doing meta things that don't work well. 

Some concerns that an educated reasonable person should raise (and have been raised are) 

1. this seems to have been made by assembling meta studies, without a lot of affirmative buy in by experts 
2. the meaning of the numbers built on survey results seems ephemeral or at least much more complicated. 
3. the claims by Stronger Minds is that they almost literally solve depression, this is dubious yet seems to follow the same framework here.

The above aren't dispositive, but the construct of WELLBY does not at all seem easy to compare to QALYs and DALYs, and the pat response is unpromising.

ben.smith @ 2022-11-27T18:06 (+6)

Can you elaborate a little bit about Stronger Minds claiming they almost literally solve depression? That would be a pretty strong claim, considering how treatment resistant depression can be.

I suppose I would be open to the idea that in Western countries we are treating the long tail of very treatment resistant depression whereas in developed countries, there are many people who get very very little of any kind of care and just a bit of therapy makes a big difference.

Making this account feels almost as bad as pulling a "Holden", @ 2022-11-27T20:29 (+1)

In this other thread, see this claim.

This phrasing is a yellow flag to me, it's a remarkably large effect, without contextualizing it in a specific, medical claim (e.g. so it can be retreated from).

The description of the therapy itself is not very promising. https://strongminds.org/our-model/

This is a coarse description. It does not suggest how such a powerful technique is reliably replicated and distributed.

Strong Minds appears to orchestrate its own evaluations, controlling data flow by local hiring contractors.

I'll be honest - personally I'm not convinced by the StrongMinds recommendation. And it all goes back to my earlier post arguing we should be researching better interventions rather than funding current ones. I was pleased to see further research on this point here and here, both of which HLI employee Joel McGuire contributed to (although in a personal capacity not related to HLI), and both of which were positive about further research.

Just imagine how much better we can do than group talk therapy. We can probably develop psychedelics / treatments to completely eradicate depression and make us all blissful all the time. Sure, it might take some time, but it will be worth it, and we'll certainly get some benefits along the way.

Relying on WELLBYs means relying on interventions we already know about. But we can do so much better than this. Consider that mental health problems used to be treated by drilling holes in people's skulls. What if we had stopped researching at that point?

I think we're missing a lot of potential value here.

If you found this post helpful, please consider completing HLI's 2022 Impact Survey.

Most questions are multiple-choice and all questions are optional. It should take you around 15 minutes depending on how much you want to say.

Extremely skeptical of this. Reading through Strong Minds' evidence base it looks like you're leaning very heavily on 2 RCTs of a total of about 600 people in Uganda, in which the outcomes were a slight reduction in a questionable mental health metric that relies on self-reporting.

You're making big claims about your intervention including that it is a better use of money than saving the lives of children. I hope you're right, otherwise you might be doing a lot of harm.