Data from the 2023 EA Forum user survey

By Sarah Cheng @ 2024-07-26T14:54 (+71)

The purpose of this post is to share data from a survey that the EA Forum team ran last year. Though we used this survey as one of many sources of information for internal analyses, I did not include any particular takeaways from this survey data in this post. I leave that as an exercise for the reader.

Overview

In August 2023, the EA Forum team ran a survey to learn more about how people use the Forum and how the Forum impacted them. We got 609 valid responses.

Thank you to everyone who responded - we really appreciate you taking the time. The results have been important for helping us understand the ways that the Forum creates value and disvalue that are otherwise hard for us to track. We’ve used it to evaluate the impact of the Forum and the marginal impact of our work, update our team’s strategy, and prioritize the work we’ve done in the past 12 months.

The person who ran the survey and wrote up the analysis is no longer at CEA, but I figured people might be interested in the results of the survey, so I’m sharing some of the data in this post. Most of the information here comes from that internal analysis, but when I use “I” that is me (Sarah) editorializing.

This post is not comprehensive, and does not include all relevant data. I did not spend time double checking any of the information from that analysis. We plan to run another (updated) survey soon for 2024.

Some Forum usage data, for context

The Forum had 4.5k monthly active and 13.7k annually active logged in users in the 12 months ending on Sept 4 2023. We estimate that the total number of users was closer to 20-30k (since about 50% of traffic is logged out).

Here’s a breakdown of usage data for logged-in users in those 12 months:

• 13.7k distinct logged in users
• 8.5k users with 3 distinct days of activity
• 5.7k users with 10 distinct days of activity
• 3.1k users who were active during ⅔ of all months
• 1.7k users who were active during ⅔ of all weeks
• 388 users who were active during on ⅔ of all days
• 4.4k distinct commenters
• 171 distinct post authors

It’s important to note that August 2022 - August 2023 was a fairly unusual time for EA, so while you can (and we have) used this survey data to estimate things like “the value the Forum generates per year”, you might think that August 2023 - August 2024 is a more typical year, and so the data from the next survey may be more representative.

Demographic reweighting^[1]

Rethink Priorities helped us with the data analysis, which included adjusting the raw data by weighting the responses to try to get a more representative view of the results. All charts below include both the raw and weighted^[2] data.

The weighting factors were:

1. Whether the respondent had posted (relative to overall Forum usage)
2. Whether the respondent had commented (relative to overall Forum usage)
3. How frequently the respondent used the Forum (relative to overall Forum usage)
4. The respondent’s EA engagement level (relative to the Forum statistics from the 2020 EA Survey)
5. The respondent’s gender (relative to the Forum statistics from the 2020 EA Survey)

Some effects of the reweighting:

• A significantly higher proportion of respondents have posted or commented on the Forum, relative to actual overall Forum usage, so reweighting decreases those percentages and the percentages of other actions (such as voting on karma).
• A plurality (around 45%) of respondents said they visit the Forum about 1-2 times a week. This is more frequent than the overall Forum population, so reweighting decreases things like the percentage of users who applied for a job due to the Forum, and the mean rating of “significantly changed your thinking”.
• Overall, respondents tended to be more highly engaged than the EA Survey population. 78.7% of respondents to our survey reported considerable or high engagement with EA, compared with 48.9% of the EA Survey. I don’t know of any significant effects this had on the reweighted data.
• A slightly larger percentage of respondents were male than compared to the EA Survey (69.9% male vs 66.2% male respectively). I believe this didn’t change the charts much, other than decreasing the mean rating of “likelihood of staying engaged with EA”.

Results

In the last year, which of the following have you done on the EA Forum?

Are you significantly engaged in any of the following cause areas or fields of work?

What proportion of the time you spend on the EA Forum do you think is useful?

We asked users how much of the time they spend on the EA Forum they think is useful, with 0 indicating 0% and 10 indicating 100%. From the free response data, it’s clear that many respondents treat the Forum as a form of entertainment, often replacing social media, Twitter, or other casual reading or listening.

Note that I think this question is rather vague. It doesn’t specify “useful” as relative to anything, doesn’t tell us to what extent the “useful” time was impactful for people, and doesn’t distinguish between “non-useful” time that is more like entertainment vs that is actually negative. All survey questions are vibes-y to some extent, and this one feels more so than the others. We plan to change how we ask about this in the next survey.

How likely are you to recommend the EA Forum to someone getting interested in EA or an EA cause area?

We asked users about their likelihood to recommend the EA Forum, with 0 indicating “Not likely at all” and 10 indicating “Extremely likely”.

In the past 12 months, how useful was the EA Forum to you as a general source of information on the EA Community?

We asked users how useful the EA Forum was as a source of information on the EA community, with 1 indicating “Not useful at all” and 7 indicating “Extremely useful”.

Notable variance:

• Daily users gave this a higher rating (6/7) vs less than once a month users (4.6/7)
• Users who selected community building, biosecurity, forecasting, and effective giving rated this question slightly higher, and users who selected global health and cause prio rated this slightly lower

In the past 12 months, how useful was the EA Forum as a source of information on specific cause areas or topics that are important to you?

We asked users how useful the EA Forum was as a source of information on specific cause areas, with 1 indicating “Not useful at all” and 7 indicating “Extremely useful”.

Notable variance:

• Daily users gave this a higher rating (5/7) vs less than once a month users (4.3/7)

In the past 12 months, how much has content or discussion on the EA Forum significantly changed your thinking on topic areas you consider important?

We asked users how much the EA Forum has significantly changed their thinking, with 1 indicating “Not at all” and 7 indicating “Extremely significantly”.

Notable variance:

• Daily users gave this a higher rating (4.5/7) vs less than once a month users (3.4/7)
• Non-male respondents gave this a lower rating (3.7/7) vs male respondents (4.0/7)
• Users who had posted or commented had a lower rating (3.7/7) vs users who hadn't (4.1/7)

In the past 12 months, how has the EA Forum affected your likelihood of staying engaged with EA?

We asked users how the EA Forum affected their likelihood of staying engaged with EA, with 1 indicating “Greatly reduced”, 4 indicating “No impact”, and 7 indicating “Greatly increased”.

Notable variance:

• Daily users gave this a higher rating (4.9/7) vs 1-2 times a month users (4.3/7)
• Users who had posted or commented had a slightly lower average rating (4.5/7) vs users who hadn't (4.8/7)
• Non-male respondents gave this a slightly lower rating (4.5/7) vs male respondents (4.6/7)

In the last year, did the EA Forum lead to any of the following career related changes for you?

We asked users if the EA Forum led to any of a list of actions and other outcomes^[3].

Notable variance:

• Compared to users who had not posted or commented, a significantly higher percentage of users who had posted applied for grants (8.3% vs 12.9%), jobs (14.8% vs 20.5%), and got job offers (2.6% vs 6.8%), whereas users who had only commented tended to apply to more jobs (19.0% vs 14.8%), but had little (<3%) effect on other career changes
• Users who didn’t post tended to change their fields and donations more frequently, suggesting they may be earlier in their career or more open to change

In the last year, how was posting or commenting valuable to you?

We asked users if posting or commenting on the EA Forum led to any of a list of outcomes^[4].

A selection from free responses^[5]

I randomly selected some rows in the spreadsheet of responses and pulled out the free response answers. I redacted identifiable information, and asked ChatGPT to rewrite them in a consistent, neutral style. Here are some excerpts from that:

• The topics and the tone of the dialogue provide a sense of the community’s priorities and norms. While some aspects are favorable, others are not. The detailed and nuanced discussions can be exhausting. Keeping up with one topic requires understanding many related topics and community dynamics, which is challenging.
• The forum is valuable, but I lack the time to write or comment.
• It’s a relatively fun and easy way to absorb new information and stay updated without becoming too isolated. I often refer to or share posts as part of my community-building and talent search efforts.
• The language barrier can deter people. A translation feature might be helpful.
• It’s useful for staying updated on news, discussions, and debates within the community. However, it’s not suitable for newcomers, as posts can be off-putting without context and overly technical.
• The most useful aspect has been reviewing different users’ posting histories to understand their perspectives, especially when evaluating them for grants.
• Finding unique ideas outside of mainstream EA.
• It helps me stay engaged and find interesting opportunities. It has slightly influenced my viewpoints but hasn’t drastically changed my priorities. I’ve considered posting but feel I have nothing worthwhile to contribute. I hope the quality and novelty of posts remain high.
• I use it for inspiration in assessing the efficiency of various ideas and identifying weaknesses in the community. However, some criticisms are too technical, and my limited understanding of statistics hinders my ability to fully support the community.
• Occasionally, good content appears, often similar to what’s found on LW. The forum offers insights into the community and helps gauge its overall sentiment.
• The community is welcoming, even though I have strong disagreements and critiques of mainstream ideas. Without this forum, in-depth critiques would struggle to reach the intended audience.
• When applying for a job at <org>, I wrote a post about <topic> to show my initiative. Although I didn't get the job, someone from <another org> reached out to me because of that post and encouraged me to apply for a position there. I wasn't planning to apply otherwise, and that's now my job.

1. ^{^}

   Weighting aims to correct for unrepresentativeness in the acquired sample by ‘reweighting’ the respondents. At a very simple level, it can be thought of like the following situation: Imagine our target population is 50:50 men and women, but our sample is 25:75 men and women. Weighting would correct for this by giving twice the weight to each man (2*25 = 50), and two thirds of the weight to each women (.67 * 75 = 50). If we then summarized data with each person contributing to the estimate according to their respective weights, it would be as if we had a sample with a 50:50 gender split.

   This is a common technique in survey data, and our specific implementation of it utilizes a widely used and standardized approach developed by the American National Election Study team to calculate weights (detailed here).

   There is also a penalty to be paid for weighting, in that there is more uncertainty around our estimates the more we have to correct/reweight the sample to make it representative. Hence, the raw estimates will generally have smaller confidence intervals than the weighted estimates, even though they have the same number of respondents contributing to them.

   The main drawback of weighting (besides a loss of precision in the estimates, which is simply a necessary price to pay) is that there is no guarantee that our estimates are fully representative of what would happen if we drew a real random sample from the population. This is because there may be other factors that we have not or cannot weight for that are also related to the outcome. In particular, in the case of the EA community, unlike a national survey, we lack official data providing information about the true composition of the population on a wide variety of characteristics, to weight towards. So, even though we have weighted according to gender, it could be that we still only have a particular type of man in the sample, and this type of man responds differently to the men who did not show up in the sample due to some factor we do not have access to. Hence, the weighted responses should be considered like a best guess at what is going on in the true population of forum users given the information we have, but we cannot be sure of its accuracy.

2. ^{^}

   Note that the labels say “weighted (freq)” and “weighted (with frequency)” because RP originally did the weighting with four factors (excluding frequency), then later did a weighting with all five factors (including frequency). I thought it would be confusing to show the intermediary bars in the charts so I asked them to generate new charts with just the raw data and final weighted data.

3. ^{^}

   The options were:

   I got interested in an organization because of the Forum, and later got involved with their work

   I applied to a job because of the Forum

   I applied to a job because of the Forum and was given an offer

   I applied to a grant because of the Forum

   I changed my area of study because of the Forum

   I changed the field I work in (or plan to work in) because of the Forum

   I changed where I donate because of the Forum

   None of the above

   Other

4. ^{^}

   The options were:

   I got personal recognition for my work

   I connected with individuals or organizations

   I was able to influence work or ideas in EA

   I got feedback

   I got answers to questions that I have

   I got applicants for projects or initiatives

   I have not posted or commented

   Other

5. ^{^}

   The two free response questions were “How, specifically, has the Forum been useful or impactful for you?”, and the “Do you have any other comments?” at the end.