What are some measurable proxies for EA community health?

By michel @ 2022-07-14T06:26 (+32)

I'm worried about EA community health. I think there are warning signs in public perceptions (e.g., Emile Torres hit-piece still being the top search result when you search google for "longtermism") and internal critiques (e.g. highly upvoted forum posts in the past year that expose negative aspects of EA).^[1]

I'd like to start having more serious discussions about how the health of the EA community will evolve. In particular, I'd like to see serious forecasting about whether there will be a flourishing EA community in the coming decade(s).

If there's a solid chance (say > 20%) that the EA community becomes toxic to associate with by 2025, I think this would be a really big deal. This would call for more strategic, community health-focused movement building to allow EA to live up to its potential, as well as possible worst-case planning for how we can still make progress on the most pressing problems in the absence of a likable EA community.

To forecast EA community health, we need measurable proxies of EA community health. Can you all help me brainstorm such proxies?

Here are some I've considered:

Number of top 20 EA university groups with greater than 10 members
Number of hit pieces (i.e., articles with an overall negative impression of effective altruism) in the following prominent news outlets: NYT, Washington Post, The Economist, Politico, Atlantic, CNN, Fox News, Forbes, NBC, Wall Street Journal, BBC, The New Yorker, The Associated Press.
Proportion of general public surveyed on Amazon MTurk who have a favorable impression of EA, conditional on them knowing about EA
# of top 20 university groups that have gotten protested or publicly mocked
Proportion of top 20 google search results for “effective altruism” that are critical of EA
And then some metrics that are more related to size, less to community health, which could help to brainstrom other proxies.
- Yearly interest in “effective altruism” on Google Trends relative to 2017 (the year with the current peak; e.g., 1.2x 2017)
- Number of subscribers on Effective Altruism subbreddit
- Proportion of top 15 forum posts every year that critique EA (adjusted for karma inflation)
- Number of people who take the EA Survey

The ideal proxy, in my mind, tracks community perception/health more than raw size. These are closely related though, so I'm not ruling out size proxies. The ideal proxy can also be tracked regularly to see how perceptions change.

If there's any consensus on the best proxy (or the best few proxies), I'll make a Metaculus question and link it here. There are already some existing metaculus questions,^[2] but I'd like to make a more community health-specific question.

^{^}
Self-criticism could also signal a healthy community, so interpreting the presence of internal critiques as a "bad omen" is shaky. But I'm confident that some of these critical posts point to real problems, and their number of upvotes point to the fact that these problems are quite apparent.
^{^}

Johan van den Heuvel @ 2022-07-14T07:41 (+12)

Would it be possible to measure "churn rate" somehow? I feel like this is a very important indicator for community health. A number of proxies you proposed have more of an outside perspective, from which it might be difficult to see the internal community health. There are of course multiple reasons for people leaving a community, but I would expect a strong correlation between more people leaving and how "toxic" EA becomes.

MichelJusten @ 2022-07-14T17:45 (+5)

Nice, I agree chrun rate seems like a good indicator of community health, but I see two problems:

1) It seems hard to measure. Most people journey through EA will be online and difficult to track, and even collecting data on initiatives like EA virtual programs seems unreliable. (I think I remember them having a low follow-up response rate).

2) Churn rate relies on people entering the community in the first place. (I understand churn rate to measure the number of people who bounce from EA after a relatively short time period (e.g. 30 days). However, I can envision worlds where community health degrades so seriously that there stops being any people joining the community in the first place who could then be detected by a churn rate indicator.

An alternative measure could be a more long term churn rate– rate of people who are already somewhat engaged with community who leave the community.

There are problems with every proxy, so just because this one has problems doesn't mean it might not be one of the best ones. Appreciate you suggesting it!

Luke Freeman @ 2022-07-15T02:40 (+5)

You can measure various different churn rates wherever they make sense and measure the average increase/decrease in churn (is churn rising, falling, staying the same).

Luke Freeman @ 2022-07-15T02:39 (+4)

I'd add sentiment analysis on public social media (twitter is pretty easy for this) for a few key terms, accounts, and hashtags.

MichelJusten @ 2022-07-15T05:47 (+1)

Ooo I hadn’t thought of this. Good idea!

Luke Freeman @ 2022-07-16T02:53 (+3)

Nice! Similarly you could look at comments and posts on the EA forum.

MichelJusten @ 2022-09-01T15:08 (+1)

Just saw this forum question from 2019 that I'll flag here: https://forum.effectivealtruism.org/posts/aEnvEmvNrhPhkTE38/what-metrics-may-be-useful-to-measure-the-health-of-the-ea

jwpieters @ 2022-07-22T11:01 (+1)

I share this concern, deeply. I think the EA community could be necessary for a lot of future impact. Risks to the community are therefore very bad.

I'm curious about the term "community health" though. Maybe there are actually two distinct categories of things to track here? One is internal - how people feel and interact within EA - and the other is external - how the general public perceives EA.