Retention in EA - Part III: Retention Comparisons

By Ben_West🔸 @ 2021-02-05T19:02 (+37)

Summary

• I spent one day trying to compare retention in EA to other movements. I was unable to find anything that I was very happy with, but I’m sharing these comparisons in case they are helpful to others. Hopefully Cunningham’s law will kick in and someone will be spurred to do a better analysis.
• I found semi-comparable estimates of retention for Southern Baptist megachurch youth ministry participants, vegetarians, and civil rights activists who participated in Freedom Summer. While none of these data sources seemed precise or perfectly comparable, my best guess is that EA is slightly better at retention than they are, and unlikely to be substantially worse.

How does EA compare to other movements?

Most interviewees remarked that, once someone gets to the point where their career is driven by EA considerations, it’s unlikely that they will leave. This might suggest that CEA should not focus on retention, as it is already going well.

For some insight on this, I tried to compare Ben Toddd’s estimates with estimates from other movements. My best guess is that EA has better retention than Southern Baptist megachurches and Freedom Summer participants, and similar or slightly worse retention to vegetarianism. Most of my uncertainty comes from how reliable these estimates are and how comparable they are to each other.

Because of how wide my error bars are, it’s hard to say too much with confidence. I don’t think EA has a severe retention problem (compared to e.g. Baptist churches). We also don’t seem substantially better at retention than others (e.g. vegetarianism). So my guess is that EA is slightly above average at retaining highly engaged members (compared to other movements), which makes me think it’s worth investing a moderate amount of resources into retention.

Ben Todd’s estimates

Taken from More empirical data on 'value drift’.

Youth Ministry Adherence

An Assessment of Dropout Rates of Former Youth Ministry Participants in Conservative Southern Baptist Megachurches classified youth ministry participants (teenagers who attended a certain type of religious education) into one of four engagement levels based on their answer to a survey (asking e.g. how frequently they attend church) and measured their retention. 

I think we are mostly interested in those listed as “engaged disciples” who continue to self-report as “high involvement”, so the 42% number is what I’m going to focus on.

This study did not control for the fact that people who were older at the time of surveying were more likely to have dropped out.[2] Also note that the survey was retrospective and presumably suffers from response bias. 

To try to handle the fact they didn’t control for age, I tried to find the parameters for a geometric distribution which fit their overall results of 42% dropout. A constant 14.5% annual dropout rate results in 42% of respondents having dropped out:

This assumes that “engaged disciples” are evenly distributed across the ages, which is probably not completely true but I would guess mostly true. A 14.5% annual dropout rate is a 54% five year dropout rate.

I’m not very confident in the 54% number, but I feel relatively confident that the number is higher than Ben Todd’s estimate of 20%. The paper cites another survey which found that 70-90% of teenagers stop attending church within two years of high school graduation. (One wrinkle is that there is some evidence people reengage with the church in their late 20s/30s, meaning that it’s a little hard to define what “dropout” actually means.)

Vegetarian adherence

ACE estimates that the average vegetarian stays vegetarian for 3.9-7.2 years,[3] implying a five-year dropout rate of 53%-77%.[4] My guess is that the average vegetarian is equivalent to Ben’s lowest level of engagement (GWWC member but not otherwise involved) or even less engaged, and being vegetarian seems about as big of a commitment as keeping the GWWC pledge. This implies a dropout rate higher than what Ben estimated for EA.

US Civil Rights Activists - Freedom Summer

Freedom Summer was a 1964 project with extremely high asks of its participants: they were mostly college students who gave up their summer to be unpaid volunteers facing severe violence (several were murdered). Participants needed to apply beforehand, and those who applied but withdrew from the project before it started give some estimate of adherence amongst civil rights activists, split by engagement level.

I think that attending Freedom Summer is a sign of involvement in the civil rights movement very roughly comparable to how having attended early EA retreats is a sign of being at one of Ben’s two highest levels of EA involvement. So that’s what I’m going to compare it to, even though the comparison is quite inexact.[5]

The Biographical Consequences of Activism followed up with participants approximately 20 years after Freedom Summer and found the following levels of engagement:

Again, the methodology makes comparisons hard, but it seems like the dropout rate from Freedom Summer is higher than Ben Todd’s estimates for EA.

Religious adherence

Social Predictors of Retention in and Switching from the Religious Faith of Family of Origin: Another Look Using Religious Tradition Self-Identification found that age was positively correlated with remaining in a religion, implying a negative dropout rate. I’m not sure whether this is because people drop out when they are young and then rejoin when they are older or if people who were born longer ago are just more committed to their religion, but either way it seems different enough from EA that the comparison isn’t useful.

Footnotes

1. In this and the rest of this section I’m going to assume a constant annual dropout rate, i.e. a geometric distribution, unless otherwise specified.
2. In the section where the author analyzes demographic variables, they say that only educational ones had a significant impact. Taken at face value, this implies that age does not correlate with dropout rate, which seems unlikely to me, but possible.
3. Taken from their “final estimate” but excluding the effect of cultured meat alternatives, death, and GCRs, since those do not seem relevant to the question I’m trying to answer here.
4. Assuming a geometric distribution whose expectation is 1/p
5. There are a lot of ways in which Freedom Summer is different from EA, the most notable being the violence participants experienced. Another major concern is that the Civil Rights Act of 1968 is seen by some as the culmination of the civil rights movement, meaning that many participants perhaps should have been expected to move to other causes after 1968 (e.g. anti-Vietnam War). But I think we would have expected many of them to be active in at least some social movement, if they were truly “retained”.
6. This is the percentage who were active in civil rights at some point within the five years following Freedom Summer, but I am simplifying this to assume it’s an annual dropout rate. My justification for this is a) there probably aren’t many people who didn’t engage for 4 years and then became active again in the fifth, and b) to the extent that these people do exist, it’s arguably more accurate to classify them as having disengaged and then reengaged instead of having been retained.

kierangreig @ 2021-02-09T00:49 (+2)

Thanks for writing this! I found it a really interesting series and kudos to you for sharing earlier-stage thinking publicly. I definitely know that I can find sharing such thinking pretty daunting! 

Small point:

>ACE estimates that the average vegetarian stays vegetarian for 3.9-7.2 years, implying a five-year dropout rate of 53%-77%. 

I think that 3.9-7.2 is their estimate for (i) the average vegetarian adherence length, but you might be interpreting that here as more like (ii) the median length of vegetarian adherence? 

From that ACE report:

> We can see in the Guesstimate model that after multiplying, the length of adherence for current vegetarians is 32–49 years.

And then they calculate the length for former vegetarians using the following as a basis:

I’d say the underlying distribution here is pretty skewed, so the difference between that average vegetarian length and median vegetarian length might be pretty significant. 

So I guess my pretty quick sense is that the median vegetarian adherence length may be a fair bit shorter than 3.9 - 7.2 years. And if you are interpreting that 3.9-7.2 as being more about the median length, and it was in fact a fair bit shorter, then that could meaningfully change some of your conclusions here.     

I think that is all somewhat nitpicky though, and I could certainly be wrong about it! Regardless, thanks again for sharing all of this. :)

Ben_West @ 2021-02-10T22:24 (+4)

Thanks Kieren! I was interpreting it to be the expectation of a geometric distribution (i.e. mean length assuming a constant annual probability of leaving), which I think is the correct way to interpret that number? Let me know if that’s wrong though!

The assumption that length is geometrically distributed might not be warranted, I'm not sure.

kierangreig @ 2021-02-11T16:14 (+4)

Thanks, Ben! That all seems fair enough for these purposes. 

Fwiw, I think that number might be more the arithmetic mean of some observations. Interpreting it as a geometric mean seems like it doesn’t strongly violate much, but I think the geometric mean is going to be a little bit lower. 

But yeah, I doubt it makes much of a difference in the scheme of things! 

Thanks again for sharing all this :)