Why does EA use QALYs instead of experience sampling?

I think ESM is the theoretically ideal measure of happiness and thus EA - indeed, everyone - should use it as the outcome measure of impact (I assume wellbeing consists in happiness).

As you laid out in this comment, it looks like experience sampling is not getting strong uptake in academia.

Here's a short argument:

• (a) Experience-sampling is theoretically the best way to measure happiness
• (b) It's feasible to build experience-sampling infrastructure, e.g. Natália's mobile app proposal
• (c) Academics & other stakeholders aren't planning to build such infrastructure
• Ergo: building experience-sampling infrastructure is a neglected, tractable, and impactful intervention that EA could undertake

First, it's not really the case that EAs use QALYs/DALYs. GWWC and GiveWell used to use them, but GWWC no longer exists as an independent entity and GiveWell now use their own metric.

This is a good point.

I think that GWWC & GiveWell's earlier use of QALYs created a lot of path dependence, such that current EA prioritization remains influenced by the QALY framework even though no organization explicitly uses it at present.

Considering an alternate timeline can help draw out the path dependence:

Imagine a world where the DCP project started in 2019, rather than 1993. In 2019, lots of people have smartphones, so experience sampling is a viable method.

The DCP researchers decide to use experience sampling instead of retrospective surveys to determine QALY weights.

Because they're using experience sampling, mental health disorders are more highly weighted in the DCP.

In this world, GiveWell gets started in 2026. GiveWell takes a look at the DCP research, and sees that mental health disorders are highly weighted. So, they decide to prioritize research into mental health interventions.

Even after GiveWell moves away from the DCP weightings (in 2036, or whenever), mental health interventions remain their main focus, because that's where they have the most granular models & the strongest network.

A minor correction: GiveWell uses DALY to measure mortality and morbidity. (Well, for malaria they actually don't look at the impact of prevention on morbidity, only mortality, since the former is relatively small -- see row 22 here.) Maybe what you had in mind is their "moral weights" which they use to convert between life years and income.

Like cole_haus points out below, ESM's results would enter disability weights (which are used to construct DALYs) to affect how health interventions are prioritized. Currently disability weights involve hypothetical surveys using methods described in cole_haus' comment, with a major issue being most respondents haven't experienced those conditions. ESM would correct that.

To use ESM results as inputs into disability weights though you'd want a representative sample. Looking at app users is a first step but you'd want to ideally do representative sampling or at least weighting. Otherwise you only capture people who would use the app. Having a large enough sample so you can break down by medical conditions is also a challenge. (For doing all these things properly, I suggest partnering with academics or at least professional researchers experienced in the relevant statistical analysis etc. Someone mentioned lack of demand from users being a potential issue -- perhaps they can be incentivized.)

Another way to solve the hypothetical bias issue is to look at surveys that include happiness metrics and

• have other characteristics of respondents
• have nationally representative samples

such as the Gallup World Poll (whose results are used in the World Happiness Report) and the World Value Survey. (Both mentioned here.) The individual-level data can be used to examine the relationship between medical conditions and happiness (this paper uses similar data to look at income and happiness, and this paper on the impact of relatives dying on happiness). I believe you can access the individual-level data through some university libraries. Though again there's the challenge of having a large enough sample size so you can break down by medical conditions, and they probably don't have detailed information on medical conditions. (Perhaps one advantage of an app is you can track someone over time, e.g. before and after a medical condition occurs, which you won't be able to do with these surveys if they don't have a panel.)

This is a great point.

From my (limited) experience witnessing poverty in the developing world, it's not clear to what extent moderate mobility problems increase risk of hunger.

They certainly don't help, but in many developing-world contexts, it seems like there's a measure of social surplus / social safety net which provides some buffer against hunger for people with chronic health problems.

I wonder if there's a paper on this...

Instead of really thinking "how do I feel right now?" it's really hard not to just think "in past situations like this I've put down '6' so I should put down '6' now".

Interesting. I've done experience-sampling to track my mood for about 3 years, and haven't noticed this dynamic. (It generally feels like I'm answering the question "how do I feel right now?")

Just another data point.

Before having a kid this worked properly...

This is a great point. I don't have children & this hasn't been a problem for me. Totally makes sense that this comes up once you're a parent.

I agree that DALY/QALY measurements aren't great either, though.

My intuition is that aggregating the results of the two methods would outperform either method individually, because they skew in different directions.

I don't really see ESM as being in opposition to QALYs. It seems like it's a method that you would use as an input in QALY weight determinations.

It can be, yes, but QALYs trade off length and quality of life, whereas ESM would only tell you about QoL/wellbeing. There would still need to be some other process (or some major assumptions) to anchor the results on the QALY (or DALY) scale.

Sorry, I wasn't trying to misrepresent you.

My story about what happened here:

• Over email, I pitched a "compare QALYs to experience-sampling" project
• You were initially skeptical about the idea
• Over the course of our correspondence, you updated to being less skeptical about the project, though not totally sold on it
• Eventually, you got busy & the correspondence stopped

Does that match your story?