A Patient's Perspective of Measuring Health Outcomes

By quinngrace @ 2024-11-28T20:46 (+18)

CW: This post delves into death, euthanasia, suffering, and discrimination against minorities verging on eugenics, which may be disturbing to some readers.

TL;DR

My main issues with current measurements for health outcomes (particularly QALYs) are:

1. They are decided by value sets from people who have not experienced the health state they are judging, which varies wildly from what people who experienced the health state judge it to be.
2. They de-value the cost-effectiveness of treatments in groups where people are very ill.
3. Maximising these measurements leads to very dark places.
4. There are no weights to address systemic issues.
5. They punish the groups in society who need health interventions the most.
6. They don't capture the full impact of health interventions. 

Possible solutions include creating a new measure and getting governments and policy-makers to adopt alternative measures.

*******

For about a year, I was bedbound. Had I not gotten better, I surely would have perished to my illness. Many of my doctors had given up hope, other than suggesting some supplements. I was in a State Worse than Dead. 

I suffered from profound fatigue that made even simple tasks almost impossible. Eventually, I even stopped eating. I slept 22 hours a day. The chest pain was constant and I struggled to breathe so severely that I ran out of breath when talking. My muscles were so weak that I couldn't get up off the floor without assistance. I struggled to remember things and my attention span dissolved. Light and sound hurt. I couldn't use my brain like I used to.

I got better, evidently. 

Now that I am better, I have more brain power for retrospection. If this was just a one-in-a-million-lost-the-birth-lottery isolated incident kinda thing, I'd move on. But it's not. It's a 1 in 77 kind of thing which it seems we're just neglecting^[1]. Why? Honestly, I'm not sure. But in all likelihood, QALYs played some part, since it is often used to measure health outcomes and the effectiveness of health interventions. 

I'm not a big fan of QALYs as a measurement for health outcomes for two main reasons:

1. It can be negative
2. It perpetuates existing inequalities

While I understand that many with the EA community already oppose QALYs, I want to drive home what the actual impact on a person's life can be and why even if we as a community move away from it, why it's important to get decision-makers on board too as a cause area.

The rest of this post will take the form of critiques of current QALY calculations interspersed with my own experiences of such.

It can be negative

One pivotal issue that has thus far not been considered in this debate is that HRQoL can not only be low, but also negative: ‘health states worse than dead’ (SWD) get assigned negative values. Extending the live of a person who lives in a SWD generates negative QALYs.

...

The QALY is defined as the arithmetic product of survival time and HRQoL. HRQoL, in turn, is determined by the health state an individual is living in. This means, ‘measuring’ QALYs usually involves two components: firstly, a set of health states; and secondly, numeric scores that reflect their respective desirability. These values are often also referred to as utilities, social values, preference-, (health-related) quality of life-, or QALY-weights. Customarily they are supposed to reflect the preferences of the general public

...

When it comes to SWD, however, the preferences of the general public are (1) ill-informed, (2) misconstrued, and/or (3) irrelevant. In the following, I shall further elaborate on these three points.

1. Ill-informed: The preferences of the general public do not correspond to patients’ evaluation of their own situations; they should not be confused with a measure of patients’ self-assessed HRQoL.

The number of people who upon hearing about my previous condition have said 'If it were me, I'd rather die' is astonishing. People so healthy that they don't even remember what it feels like to have a common cold are harshly judging my quality of existence. I can't blame them, a few years back when researching euthanasia for debating, I remember reading about some cases and thinking 'Wow, if that were me, I'd want to be euthanised for sure'

But imagining an experience and living it are galaxies apart.

When ill, I remember reading about MAID in Canada (a euthanasia program). People who were healthier than me had qualified for it, so had I been Canadian I would easily have qualified. It's terrifying to think that had I been born in Canada, my only 'treatment' option would have been death.^[2] Not even at my most ill did I wish for death, all I wanted was to feel like myself again. In fact, I read a study that claimed that the average lifespan of someone with my condition was about 47 years and actually lamenting the loss of life. 

If I had to choose between being severely ill for a year or going back in time to high school for a year, I'd take being severely ill hands down. Yes, I hated high school that much and yes, there are things worse than a SWD (in my opinion).

2. Misconstrued: Social value sets do not reflect the general public’s preferences for the allocation of resources.

Participants in health valuation studies are not asked how they prefer resources to be allocated. This would require using a method like the person trade-off, for example, in which participants are asked to make choices about two groups of people, which differ in size and in their health states [37]. Instead, TTO or SG are used, which ask participants to imagine being in a particular health state themselves. Yet, this one type of preferences can not easily be translated into another. Some people may, for example, say that they would rather prefer to be dead, than to be confined to bed [38]. Yet, the very same people will probably consider their preferences being misrepresented, if they led to the evaluation that people who are confined to bed should not be offered life-extending treatments. They may rightly object that this is just not what they meant.

If someone took my statement about my experience of high school being worse than a SWD and used that to justify not improving those high schools, I would most certainly object. Even if we wanted to allocate resources based on societal preferences, QALYs are not the way to do that (nor does it pretend to be).

3. Irrelevant: Even if social value sets would accurately reflect the general public’s preferences, in the context of SWD, those should be considered irrelevant.

It seems rather improbable that members of the general public in the UK, or anywhere else for that matter, would actually support the concept of SWD and their implicit value judgement—which we will discuss in more detail in the next section. However, even if some individuals wanted some other individuals to die earlier rather than later, those preferences should be deemed irrelevant for treatment reimbursement decisions.

While everyone has, of course, the right to consider their own life in a certain health state to be worse than dead and to refuse life-extending treatments, considering someone else’s life in a certain health state worse than dead is morally a completely different issue. To then also prefer that life-extending treatments are withheld from certain (other) individuals, because one prefers them to be dead, would undoubtedly be reprehensible. It would constitute an objectionable preference [11, 23].

(Emphasis added)

The end result of negative QALYs, is that treatments that otherwise would be considered cost-effective, are no longer evaluated as cost effective. Schneider goes into great detail to illustrate this in the above-quoted paper, which I recommend reading in its entirety. The darker flipside to this, is that letting people in SWD die generates QALYs.^[3]  That would surely create perverse incentives to kill off sick people. But that's just conspiracy talk surely? No one would actually argue in favour of that, right? 

In this paper, we propose and defend three economic arguments for permitting assisted dying. These arguments are not intended to provide a rationale for legalising assisted suicide or euthanasia in and of themselves; rather, they are supplementary arguments that should not be neglected when considering the ethics of assisted dying. The first argument is that permitting assisted dying allows consenting patients to avoid negative quality-adjusted life years, enabling avoidance of suffering. The second argument is that the resources consumed by patients who are denied assisted dying could instead be used to provide additional (positive) quality-adjusted life years for patients elsewhere in the healthcare system who wish to continue living and to improve their quality of life. The third argument is that organ donation may be an additional potential source of quality-adjusted life years in this context. We also anticipate and provide counterarguments to several objections to our thesis. Taken together, the cumulative avoidance of negative quality-adjusted life years and gain in positive quality-adjusted life years suggest that permitting assisted dying would substantially benefit both the small population that seeks assisted suicide or euthanasia, and the larger general population. As such, denying assisted dying is a lose–lose situation for all patients.

Source

A healthcare metric should not be designed in such a way that killing a group of people to prevent them from using further health resources and then extracting their organs is evaluated as a good thing. 

In fact, the authors have to specify:

...we stress that avoiding suffering and respecting autonomy remain the primary arguments for assisted dying; 

The fact that this needs to be added because using only QALYs as an evaluation metric would generate a dystopian nightmare, is the very problem.

All of this can be avoided by removing the concept of SWD from QALYs/not having negative QALYs. My next gripe with QALYs cannot be so easily solved.

It perpetuates existing inequalities

I love watching medical dramas because they offer a fantasy to escape into, one where doctors will go to any length to solve your medical mystery. When you have anything even vaguely unusual, trying to get effective treatment is very difficult.

(Just a note, doctors are only human. They err. In fact, a lot of these mistakes come from systemic problems and biases.)

Things I have been told by actual doctors: 

"The best treatment we know of is time, let's see how you are after 5 years." - For a severely disabling condition, that it turns out sooner treatment makes a huge difference for.

"That's not a real diagnosis, who said you have that?" About another specialist's diagnosis.

"It's probably just anxiety." Then offered no treatment or advice for my probable anxiety. Plot twist, it wasn't anxiety. This happened multiple times.

"Maybe the reason you feel so sleepy is because you aren't trying hard enough to stay awake." - It couldn't possibly be a common side effect of the medication I was taking, which immediately disappeared when I stopped it. /s

"You're feeling suicidal? Stop being such a drama queen!"

For a significant portion of medical research, women are excluded and we keep seeing the knock-on effects. Compounded by this issue are illnesses that mostly affect women, especially autoimmune diseases (especially EDS, fibromyalgia, CFS/ME, MCAS). They have been around for ages, but sexism in the medical field meant that these diseases are ignored by doctors and researchers. 

Furthermore, even when women aren't being ignored or excluded, the sex-based differences aren't adjusted for, creating unfair comparisons.

One domain common to all of the typically used health utility instruments and instruments with which readers might be more familiar such as the SF-36 (Ware & Sherbourne, 1992) is pain.1 Men and women have been found to respond differently to similar pain stimuli (Kimble et al., 2003) and to report pain differently (Soetanto, Chung, & Wong, 2004). One possible result of these differences is that men and women who report the same level of pain may have different levels of health utility. If women experience greater dysfunction at the same level of pain as reported by men, the effect of relieving the pain among women will be underestimated when using societal preferences, even if it is partially picked up by other functional domains. Another possible gender difference arises if men and women report different levels of pain when the objective pain stimulus is identical (Jackson, Iezzi, Chen, Ebnet, & Eglitis, 2005). If women report higher levels of pain for the same stimulus, relieving the pain will lead to a greater improvement in health utility for women than for men. If a treatment could completely relieve the pain and a societal preference weighted instrument were used to score health utility, this would result in an underestimate of the actual gain for women and a less favorable cost-effectiveness ratio than if a gender-specific health utility measure were used.

Source

If a measure doesn't take systemic bias into account, it will perpetuate it. (The mechanism for how this happens with QALYs will follow.)

Getting to those doctors, when I was really sick, was a mission. On good days, I could use a wheelchair, but I can't predict when those days would be in advance to reschedule the appointment if need be. Most specialist doctors don't do home visits either, so I had to come to them, in the process making myself sicker.

Strangely, most healthcare isn't designed for sick or chronically ill people. It's designed for healthy people, which is completely counterintuitive. But the more you actually need healthcare, the more barriers you'll face trying to access it. When you are really, really sick, it often morphs into a disability because the illness starts to impede daily activities.  

What if you need a wheelchair? Is your transport and the place you are going to wheelchair accessible? What if you are too sick to get out of bed? Will your doctor make a home visit or do telemedicine? Can you reschedule if you are too sick that day and it would take too much of a toll? Does the doctor know how to deal with multiple medication interactions? Does the doctor know how to deal with co-morbidities or do you need another doctor? If you can't see, are there braille and screen readers? Can appointments only be made through phone call or are online bookings an option as well, especially for highly anxious people, autistic people, and people with difficulty speaking. 

There's also a theme developing here, chronically ill and disabled people face numerous barriers to getting treatment^[4]. 

The result is not getting necessary treatment and having worse outcomes, resulting in earlier death. 

This creates a negative feedback loop because having a lower life expectancy means that person can't generate as many QALYs as a healthy and/or able-bodied counterpart. So a treatment that improves their quality of life is less likely to get funded because it can't generate as many QALYs as a treatment that treats conditions that improves the quality of life of people with long lifespans. The same barriers still continue to exist. This phenomenon is not small, in fact, the average amount having a disability shortens lives by is decades in LMIC.

As Andrade puts it:

... QALY sets up a rigid social hierarchy, a caste system, so to speak. At the top are those who are eligible for treatment options, and who receive (in the form of subsidies and taxpayers’ funds) privileges from the entire society. At the bottom are those who are not eligible for treatment, and therefore, their lives are cut short. Viciously, this very fact further contributes to their illegibility for treatment, as a shorter life expectancy makes the QALY score ever lower. Ultimately, with QALY, not all people are created equal. They may be formally equal before the law, but some are certainly not equal in terms of their worth to public health officials.

The fundamental ethical shortcoming of QALY is that in its orientation toward numbers, it fails to value life itself. Consider two treatments that use the same amount of funds. In treatment A, the life of a 35-year-old man with a rare disease is saved. In treatment B, the erectile dysfunction of 200 35-year-old patients is successfully treated. Given the large amount of people in scenario B, QALY will probably be higher. Yet, it seems more sensible to admit that even with reduced QALY, choosing A is more ethically appropriate, as a life is being saved.

...

Since QALY relies on life expectancy, young people are privileged with this criterion. And the quality-of-life years also play a role, as young people are also significantly more likely to fully recover from interventions.

This has had the practical implications. For example, NICE has pursued a policy of not including drugs for dementia on the grounds that they are too expensive and do not provide sufficient QALYs, as compared to other treatments, to which funds are allocated.

...

QALY’s discrimination is also salient in terms of race and ethnicity. Groups with higher life expectancy receive preferential treatment, given that they are in a better position to maximize QALY. ²⁷ This preferential treatment often falls along racial lines, with whites being offered treatment ahead of ethnic minorities. ²⁸ To the extent that life expectancy is shorter among ethnic minorities, ²⁹ they are excluded from the priorities.

However, it has been firmly established by research that social variables play a major role in life expectancy,^{30 ,31 , 32 , 33 , 34} and oppression and discrimination are central in this regard. Therefore, a vicious circle comes into effect. Systemic racism in society shortens life expectation of disadvantaged ethnic and racial groups. As per QALY standards, reduced life expectancy takes away priority in the allocation of resources. ³⁵ Lack of medical assistance increases stress. ³⁶ and further decreases life expectancy,³⁷ causing minority ethnic groups to be further displaced in the order of priorities in QALY allocation.

There are other reasons to question these sorts of measurements

As mentioned in this post about the HALY+ and this post about HALYs in general:

1. They neglect non-health consequences of health interventions.
2. They rely on poorly-informed judgements of the general public.
3. They fail to acknowledge extreme suffering (and happiness).
4. They capture some but not all spillover effects.
5. They are of little use in prioritising across sectors or cause areas.

Both posts are worth reading in their entirety, especially if the reader is interested in the more technical aspects of measuring health outcomes.

I'd like to just briefly touch on point 1. Measured in QALYs/DALYs, my personal improvement in health is only a fraction of the result of getting treated. (Tooting my own horn incoming): I now earn an income which I have publicly pledged to donate a portion of for the rest of my life, where I would previously be dependent on others. I also work in a high impact career deriving insights from digitising clinics in rural Rwanda.  None of this is captured by current HALY frameworks, even though they are intertwined with my health state. 

In summary, my main issues with current measurements for health outcomes (particularly QALYs) are:

1. They are decided by value sets from people who have not experienced the health state they are judging, which varies wildly from what people who experienced the health state judge it to be.
2. They de-value the cost-effectiveness of treatments in groups where people are very ill.
3. Maximising these measurements leads to very dark places.
4. There are no weights to address systemic issues.
5. They punish the groups in society who need health interventions the most.
6. They don't capture the full impact of health interventions. 

What now?

1. We should create a new metric that doesn't fall prey to the above issues. At the very least, it should take patient experience input with weights for systemic issues and have death as the lowest point on the scale, so that no person is more valuable dead than alive.
2. We should prioritise getting governments and policy-makers to move away from QALYs in favour of alternatives. 

Image source

1. ^{^}

   This source uses QALYs to compare different diseases to their relative funding, which is a measurement I criticise heavily throughout this post. However, this source is still useful to show how even under the QALY framework, ME/CFS is a neglected cause in global health and development.

2. ^{^}

   This policy has come under intense fire from disability activists. Here are some good analyses of why: https://www.aljazeera.com/opinions/2024/2/16/canadas-assisted-dying-regime-should-not-be-expanded-to-include-children https://www.cambridge.org/core/journals/palliative-and-supportive-care/article/realities-of-medical-assistance-in-dying-in-canada/3105E6A45E04DFA8602D54DF91A2F568

3. ^{^}

   Take the example of a QALY value of -0.2 placed on someone's health state. If they died 10 years prematurely, they would go up to 0, gaining 2 QALYs. Hopefully, this illustrates the danger of negative QALYs.

4. ^{^}

   World Health Organisation even highlights it here: https://www.who.int/news-room/fact-sheets/detail/disability-and-health