8 Counter-Intuitive Considerations for Psychotherapy Interventions

By John Salter @ 2024-11-11T10:53 (+75)

I'm the founder of Overcome, an EA-aligned mental health charity. Here are some counter-intuitive things I've learnt over the past 5 years.  My target audience are those hoping to build or fund a highly cost-effective psychotherapy charity. 

Treating people as well as a professional therapists is not hard.

People with nothing more than a high-school diploma and a month long crash course can treat PTSD ~75% as well as a professional therapist. Same with depression, anxiety and most other mental illnesses. Multiple systematic reviews attest to this. 

1. We give our volunteer coaches 3 weeks of training. Here's how their results compare against professional therapists in treating insomnia:

   

2. It's not just for insomnia. Here are our results across all conditions (n>300):

   

3. We were getting these results for hundreds of people even when our yearly budget was $50k and we'd only been treating people for 12 months. Others could likely replicate our results. EA approaches to mental health are grossly underfunded relative to their marginal impact.

How is it possible that non-therapists are just as good?

1. Therapists have many other competing incentives besides just making their clients feel better:
   1. Therapist get paid per session, not per successful treatment. Attracting people looking for a permanent therapist will make you much more money than continuously attracting new people, treating them, and having them leave.
   2. The demographic a therapist serves has a much larger role on their income than how well they treat people. Moving to higher paying clients who are more likely to recommend you to others is much more profitable
   3. To get more and better paying clients, you need to conform clients preferences. Sadly, clients do not know how to choose effective therapists.
2. Most important of all, the experience of the therapist barely affects psychotherapy outcomes.

Client > Relationship > Method > Therapist

Neglectedness -> Lower Tractability

The case that a dollar goes farther when spent on someone poorer, because they have more important needs yet to be fulfilled, is a great heuristic. I don't think it applies cleanly to mental health. 

• The number one factor affecting psychotherapy client outcomes is the commitment and competence of the client. This ought be intuitive as the client does~90% of the work! People in LMICs tend to have other problems that prevent them from successfully executing.
• Most organisations that grow quickly and sustainably without massive ad spend do so with referrals. Likely due to greater stigma, despite most of clients coming from low and middle-income countries (LMICs) ~70% of our referrals have come from people in high-income countries.

• Counter-intuitively, it can be more expensive to treat people in lower cost of living areas. We supported both native Kenyans (via the Shamiri Institute) vs EA charity founders (via Charity Entrepreneurship / AIM). 

  1. Kenyan vs founders
     1. Effort to secure partnership: about the same
     2. Sign-up rate: ~10% vs ~33%
     3. Sign-up form conversion rate: ~50% vs >90%
     4. Completion rate given that they attend first session: 40% vs >90%
     5. Completion rate given that they signed up: ~20% vs <81%
     6. Life satisfaction gain (0-10 scale): 2 points vs 1.5 points
     7. In terms of management time  per successful treatment, it took 4x more effort per Kenyan than per founder.
     8. Per unit effort, we were able to better results for founders than we were for native Kenyans.

  To summarise, in physical health the intervention provider usually does 90% of the work. In mental health, the client does 90% of the work. Unfortunately, these clients often lack the resources needed to execute on that work reliably. They're also less likely to refer you other people due to stigma, so it's harder to cost-effectively grow your service.

  Key takeaways: 

  - Western demographics aren't strictly worse to treat than non-western demographics because results and referrals come easier.

  - You ought attribute more of a therapy interventions results to the demographic they choose than how well the service is run

  - Getting good results in lower-income countries is harder, so you ought judge their results more favorably. See Vida Plena & Kaya Guides.

  Aiming at cheaper > Aiming at better

  It's dramatically easier to shrink your cost per user by 90% than it is to make your treatment reduce 10x more symptoms. Both have the same impact on cost-effectiveness. For this reason:

  1. Ignore things that don't affect client outcomes. Focus on:
     1. Choosing the right demographic of client to serve
     2. Selecting likable and deeply empathetic counselors (regardless of their qualifications / experience). Use roleplay assessments to test their ability to build strong rapport quickly
     3. Take the clients goals and preferences seriously. Ideally, create a bespoke plan for each client. If you can't do that, then focus your marketing on attracting only clients who will resonate with your treatment. You should be tracking which advert has attracted a given client, and change your marketing to reflect what you learn.
     4. Pairing people up to the right counselor and treatment for their goals and preferences.
  2. Recruit staff from lower cost of living areas
     1. For example, a recent graduate and a virtual assistant from Kenya administer almost all of our training to a monthly cohort of ~15 people for less than $1,000 per month.
  3. Automate everything you can

  Common problem + high quality solution + free =\= high demand

  You must also consider:

  1. How likely are clients to seek help? (most don't)
  2. How easy are they to target with ads? How likely are they to convert if they see your ad?
  3. How likely are they to recommend your services to others? How likely are those people to act on the recommendation?
  4. How accurately can users who have never heard of your service evaluate it?
  5. How long has the service been going? (the more former clients you have, the more referrals you're likely to get, the higher you'll rank on google, the better tailored you marketing will be, etc)

Be skeptical of low completion-rate therapies (<5%) regardless of apparent cost-effectiveness

1. Key point: In mental health, spontaneous remission treats more people than all other therapies combined. Low completion-rate therapies take credit for recoveries that were actually just due to spontaneous remission, thus inflating their apparent effectiveness, while having a bunch of other downsides
2. Most costs are inversely proportional to completion-rate (e.g. if it costs you $10 to recruit a user, and you retain only 5% of them, then your marketing cost per retained user is $10/0.05 = $200. For contrast, the cost of delivering Overcome's intervention is ~$35 per completing user.
3. People quitting your therapy is morally expensive
   1. They become less trusting of other therapies
   2. It wastes their time, energy, money
   3. It erodes the trust between the client and whomever referred your therapy to them

"Doing many things mediocrely" > "Doing one thing well"

1. Treating people cheaply is not that hard.  At least for the first few years or so, there's good odds you'll be constrained by demand rather than supply even if your service is free. Marketing will thus consume a lot of your time, energy, and money unless you can get referrals. If you can treat most mental illnesses, the rate at which you get appropriate referrals is likely 3-5x higher.

   

2. Most people with mental illnesses have more than one, and each interferes with the treatment of the other.
3. If you're treating people cost-effectively, what you're doing is probably out of the ordinary and hard to convince people to try.
4. Further, if you're successfully treating people cheaply using a scalable intervention, marketing is likely to cost ~30% of your total budget unless you can grow organically using referrals or secure a huge partnership.

Selection > Training

The majority of the variance in therapy outcomes that charities can both control and explain (outside of choosing the right demographic) relates to the relationship between the counselor and the client. In our experience of selecting and training >150 counselors, personality plays a far larger role than skill / training. We’ve rejected people with PhDs in favor of those without degrees because the latter were more likable, empathetic, and intelligent

Summary

1. Choice of demographic is the single most important thing to get right
2. While low-income demographics are more neglected, they are generally harder to treat effectively and harder to recruit.
3. Marketing will be hard even if your service is free and you have a broad demographic.
4. Focus on how counselors were selected, not their CVs / qualifications.
5. Do not assume counselors will get significantly better with experience.
6. Be wary of low completion-rate therapies
7. Reducing costs is the dominant strategy to increase cost-effectiveness.

geoffrey @ 2024-11-11T14:31 (+5)

Hey John, this is very cool to read. I like the focus on what surprised you as a founder (and maybe newcomer?) in the mental health field.

I'm curious to hear more about the implementation details. Could you tell me more about the length, intensity, and duration of a typical treatment program? I saw 6 sessions in a graph which makes me think this is once-a-week program for 1-2 hour sessions over 1-2 months

Less sessions is a reliable way to reduce cost, but my understanding is there’s a U-shaped curve to cost-effectiveness here. 1 session doesn't have enough benefits but 100 sessions costs too much and doesn't add more benefit.

Also, are you targeting specific conditions? I see improvement in insomnia but that can arise from a sleep intervention or a general CBT course too

John Salter @ 2024-11-11T15:02 (+4)

and maybe newcomer?

I am! Just under two years delivering psychotherapy interventions, ~5 years in mental health more generally

Could you tell me more about the length, intensity, and duration of a typical treatment program?

We offer a minimum of six weeks, with no arbitrary cap. It's once (or rarely twice) a week for ~1 hour at a time. I'd suggest that six weeks is the most cost-effective if you are limited by supply, but in practice it tends to be longer because often you have spare capacity.
 

Less sessions is a reliable way to reduce cost, but my understanding is there’s a U-shaped curve to cost-effectiveness here. 1 session doesn't have enough benefits but 100 sessions costs too much and doesn't add more benefit.

That sounds about right. 

 

Also, are you targeting specific conditions? I see improvement in insomnia but that can arise from a sleep intervention or a general CBT course too

Depends on the client. Mostly our counselling is bespoke, but we have some programmes for more specialised issues (e.g. chronic insomnia, addiction, phobia)
 

Julia_Wise🔸 @ 2024-11-13T14:54 (+4)

Glad to see more attention on this area!
A little spot-checking:
"People with nothing more than a high-school diploma and a month long crash course can treat PTSD ~75% as well as a professional therapist." The metastudy linked doesn't attempt to compare lay counselors with professional therapists; it's only about trained lay counselors.

John Salter @ 2024-11-13T15:39 (+2)

Thank you for thinking critically about my work! You're right, it is not a direct comparison.

It shows an effect size of just over 0.6. The typical for most psychotherapies is 0.8. (see the Perplexity.ai summary below of the PTSD meta-analyses in the literature)

 

I did 0.6 / 0.8, which is 0.75. That equates to 75%. 

As this is pretty approximate, especially given that it didn't directly compare the same groups against one and other. I included the ~ before 75% to show that it shouldn't be used as a precise figure.  In hindsight, I regret not making this more explicit. 

That being said, the near equivalency between laypersons and trained therapists is widely accepted. Every single EA mental health charity uses laypersons rather than professional therapists for this reason