The 2014 Survey of Effective Altruists: Results and Analysis

Thank you for doing this survey and analysis. I regret that the feedback from me was primarily critical, and that this reply will follow in a similar vein. But I don’t believe the data from this survey is interpretable in most cases, and I think that the main value of this work is as a cautionary example.

A biased analogy

Suppose you wanted to survey the population of Christians at Oxford: maybe you wanted to know their demographics, the mix of denominations, their beliefs on ‘hot button’ bioethical topics, and things like that.

Suppose you did it by going around the local churches and asking the priests to spread the word to their congregants. The local catholic church is very excited, and the priest promises to mention at the end of his sermon; you can’t get through to the Anglican vicar, but the secretary promises she’ll mention it in the next newsletter; the evangelical pastor politely declines.

You get the results, and you find that Christians in Oxford are overwhelmingly catholic, that they are primarily White and Hispanic, and tend conservative on most bioethical issues, and are particularly opposed to abortion and many forms of contraception.

Surveys and Sampling

Of course, you shouldn’t think that, because this sort of survey is shot through with sampling bias. You’d expect Catholics are far more likely to respond to the survey than evangelicals, so instead of getting a balanced picture of the ‘Christians in Oxford’ population, you get a picture of a ‘primarily Catholics in Oxford with some others’ – and predictably the ethnicity data and the bioethical beliefs are skewed.

I hope EA is non-denominational (or failing that, ecumenical), but there is a substructure to the EA population – folks who hang around LessWrong tend to be different from those who hang around Giving What We Can, for example. Further they likely differ in ways the survey is interested in: their gender, their giving, what causes they support, and so on. To survey of ‘The Effective Altruism Movement’, the EAs who cluster in both need to be represented proportionately (ditto all the other subgroups).

The original plan (as I understand) was to obviate the sampling concerns by just sampling the entire population. This was highly over-confident (when has a voluntary survey captured 90%+ of a target population?) and the consequences of its failure to become a de facto ‘EA census’ significant. The blanket advertising of the survey was taken up by some sources more than others: LessWrong put in on their main page, whilst Giving What We Can didn’t email it around – for example. Analogous to the Catholics and the Pentecostals, you would anticipate LWers to be significantly over-sampled versus folks in GWWC (or, indeed, versus many other groups, as I’d guess LW’s ‘reach’ to its membership via its main page is much better than many other groups). Consequently results like the proportion of EAs who care about AI/x-risk, where most EAs live, or what got them involved in EA you would predict to be slanted towards what LWers care about, where LWers live (bay area), or how LWers got involved in EA (LW!).

If the subgroups didn’t differ, we could breathe a sigh of relief. Alas, not so: the subgroups identified by URL significantly differ across a variety of demographic information, and their absolute size (often 10-20%) makes the difference practically as well as statistically significant – I’d guess if you compared ‘where you heard about EA’ against URL, you’d see an even bigger difference. It may understate the case – if one moved from 3 groups (LW, EA FB, contacts) to 2 (LW, non-LW), one may see more differences, and the missing variable issues and smaller subgroup size mean the point estimates for (e.g.) what proportion of LWers care about X-risk is not that reliable.

Convenience sampling is always dicey, as unlike probabilistic sampling any error in parameter estimate due to bias will not expectedly diminish as you increase the sample size. However, the sampling strategy in this case is particularly undesirable as the likely bias runs pretty much parallel to the things you are interested in: you might hope that (for example) the population of the EA facebook might not be too slanted in terms of cause selection compared to the ‘real’ EA population – not a group like GWWC, LW, CFAR, etc.

What makes it particularly problematic is that it is very hard estimate the ‘size’ of this bias: I wouldn’t be surprised if this survey only oversampled LWers by 5-10%, but I wouldn’t be that surprised if it oversampled LWers by a factor of 3 either. The problem is that any ‘surprise’ I get from the survey mostly goes to adjusting my expectation of how biased it is. Suppose I think ‘EA’ is 50% male and I expect the survey to overestimate the %age male by 15%. Suppose the survey said EA was 90% male. I am going to be much more uncertain about the degree of over-representation than I am about what I think the ‘true EA male fraction’ is. So the update will be to something like 52% male and the survey overestimating by 28%. To the extent I am not an ideal epistemic agent, feeding me difficult to interpret data might make my estimates worse, not better.

To find fault is easy; to plan well, difficult

Science rewards caution and planning; many problems found in analysis could only have been fixed in design, and post-hoc cleaning of data is seldom feasible and still seldomer easy. Further planning could have made the results more interpretable. Survey design has a variety of jargon like “population definition”, “sampling frame”. More careful discussion of what the target population was and how they were going to be reached could have flagged the sampling bias worry sooner, likewise how likely a ‘saturation’ strategy was to succeed. As it was most of the discussion seemed to be focused on grabbing as many people as possible.

Similarly, ‘baking in’ the intended analysis plan with the survey itself would have helped to make sure the data could be analysed in the manner intended (my understanding – correct me if I’m wrong! – is that the planning of exactly what analysis would be done happened after the survey was in the wild). In view of the sampling worries, the analysis was planned to avoid giving aggregate measures sensitive to sampling bias, but instead explore relationships between groups via regression (e.g. what factors predict amount given to charity). However, my understanding is this pre-registered plan had to be abandoned as the data was not amenable. Losing the pre-registered plan for a new one which shares no common elements is regrettable (especially as the new results are very vulnerable to sampling bias), and a bit of a red flag.

On getting better data, and on using data better

Given the above, I think the survey offers extremely unreliable data. I’m not sure I agree with the authors it is ‘better than nothing’, or better than our intuitions - given most of us are imperfect cognizers, it might lead us more astray to the ‘true nature’ of the EA community. I am pretty confident it is not worth the collective time and energy it has taken: it probably took a couple of hundred hours of the EA community’s time to fill in the surveys, leave alone the significant work from the team in terms of design, analysis, etc.

Although some things could not have been helped, I think many things could have, and there were better approaches ex ante:

1) It is always hard to calibrate one’s lack of knowledge about something. But googling things like ‘survey design’, ‘sampling’, and similar are fruitful – if nothing else, they suggest that ‘doing a survey’ is not always straightforward and easy, and put one on guard for hidden pitfalls. This sort of screening should be particularly encouraged if one isn’t a domain expert: many things in medicine concord with common sense, but some things do not, likewise statistics and analysis, and no doubt likewise many other matters I know even less about.

2) Clever and sensible the EA community generally is, it may not always be sufficient to ask for feedback on a survey idea and then interpreting the lack of response as a tacit green light. Sometimes ‘We need expertise and will not start until we have engaged some’, although more cautious, is also more better. I’d anticipate this concern will grow in significance as EAs tackle things ‘further afield’ from their background and training.

3) You did get a relative domain expert raise the sampling concerns to you within a few hours of going live. Laudable though it was that you were responsive to this criticism and (for example) tracked URL data to get a better handle on sampling concerns, invited your critics to review prior drafts and analysis, and mention the methodological concerns prominently, it took a little too long to get there. There also seemed a fair about of over-confidence and defensiveness – not only from some members of the survey team, but from others who thought that, although they hadn’t considered X before and didn’t know a huge amount about X, that on the basis of summary reflection X wasn’t such a big deal. Calling a pause very early may have been feasible, and may have salvaged the survey from the problems above.

This all comes across as disheartening. I was disheartened too: effective altruism intends to put a strong emphasis on being quantitative, getting robust data, and so forth. Yet when we try to practice what we preach, our efforts leave much to be desired (this survey is not the only – or the worst – example). In the same way good outcomes are not guaranteed by good intentions, good information is not guaranteed by good will and hard work. In some ways we are trailblazers in looking hard at the first problem, but for the second we have the benefit of the bitter experience of the scientists and statisticians who have gone before us. Let us avoid recapitulating their mistakes.

Thanks for running the survey, writing it up, and posting the data. I think this is chiefly valuable for giving people an approximate overview of what we know about the movement, so it's great to have the summary document which does that.

I would have preferred fewer attempts to look for statistical significance, as I'm not sure they ever helped much and think they have led you to at least one misleading conclusion. In particular:

Reading the “The Four Focus Areas of Effective Altruism”, one would expect a roughly even split between (1) poverty, (2) metacharity, (3) far future / x­risk / AI, and (4) nonhuman animals. Above, instead of equal splits, poverty emerges as a clear leader [Footnote: Statistically significant with a t­-test, p < 0.0001]

On the contrary, I think the main message from the data is that in the sample collected, they are roughly evenly split. The biggest of the four beats the smallest by less than a factor of two -- this is a relatively small difference when there are no mechanisms I can see which should equalise their size (I would not have been shocked if you'd found an order of magnitude difference between some two of them).

Doing a test here for statistical significance is basically checking the hypothesis that survey participants were drawn from a distribution with exactly equal proportions supporting the four causes. But that's obviously nonsense -- we don't need a big survey to tell us that. It does tell us that poverty is the biggest (where we might not have been confident about which was), but statistical significance is misleading in terms of what that means -- the raw ratios are more informative.

Thank you to the survey team for completing what is an easy-to-underestimate volume of work. Thank you also to the many who completed this survey, helping us to both understand different EA communities better and to improve this process of learning about ourselves as a wider group in future years.

I have designed and analysed several consumer surveys professionally as part of my job as a strategy consultant.

There is already a discussion of sample bias so I will leave those issues alone in this post and focus on three simple suggestions to make the process easier and more reliable for when this valuable exercise is repeated next year.

Firstly, we should use commercial software to operate the survey rather than trying to build something ourselves. These are both less effort and more reliable. For example, SurveyMonkey could have done everything this survey does for about £300. I'm happy to pay that myself next year to avoid some of the data quality issues.

Secondly, we should use live data validation to improve data collection, data integrity and ease of analysis. SurveyMonkey or other tools can help John to fill in his age in the right box. It could refuse to believe the 7 year old, and suggest that they have another go at entering their age. It could also be valuable to do some respondent validation by asking people to answer a question with a given answer, removing any random clickers or poor quality respondents who are speeding through (eg "Please enter the number '2' in letters into the textbox to prove you are not a robot. For example, the number '1' in letters is 'one'")

Thirdly, we should do more testing by trying out draft versions with respondents who have not written the survey. It is very, very hard to estimate how people are going to read a particular question, or which options should be included in multiple choice questions. Within my firm, it is typical for an entire project team to run through a survey several times before sending it out to the public. Part of the value here is that most team members were not closely involved in writing the survey, and so won't necessarily be reading it in the way the author expected them to read it. I would suggest you want to try any version of the survey out with a large group (at least twenty) of different people who might answer it, to catch the interpretations of questions which different groups might have. Does the EA affiliation filter work as hoped for? Are there important charities which we should include in the prompt list? It does not seem unreasonable to pilot and redraft a few times with a diverse group of willing volunteers before releasing generally.

The analysis throws up several interesting conclusions, and I have learned a lot by reading through it. The main shocks are: the relatively low levels of donations in $ terms by many self-identified EAs, the relatively low proportion of EAs identifying chapters/local groups as a reason for joining or identifying with the community and, (for me) the encouragingly high proportion of respondents who are vegetarian or vegan.

I'm going to set aside some time in May to go through the data in a 'consulting' sort of way to see if that approach throws up anything interesting or different to others and will circulate with the survey team before publishing here.

Thanks for this, and thanks for putting the full data on github. I'll have a sift through it tonight and see how far I get towards processing it all (perhaps I'll decide it's too messy and I'll just be grateful for the results in the report!).

I have one specific comment so far: on page 12 of the PDF you have rationality as the third-highest-ranking cause. This was surprisingly high to me. The table in imdata.csv has it as "Improving rationality or science", which is grouping together two very different things. (I am strongly in favour of improving science, such as with open data, a culture of sharing lab secrets and code, etc.; I'm pretty indifferent to CFAR-style rationality.)

"238 EAs in our sample donated 1% of their income or more, and 84 EAs in our sample give 10% of their income."

I was surprised by this. In particular, 22% (127/588) of people identifying as EAs do not donate. (Of course they may have good reasons for not donating, e.g. if they are employed by an EA charity or if they are currently investing in order to give more in the future). Do we know why so many people identify as EAs but do not presently donate?

I've made a bar chart plotter thing with the survey data: link.

PDF link doesn't exist anymore. @Peter_Hurford

The first 17 entries in imdata.csv have some mixed-up columns, starting (at latest) from

Have you volunteered or worked for any of the following organisations? [Machine Intelligence Research Institute]

until (at least)

Over 2013, which charities did you donate to? [Against Malaria Foundation].

Some of this I can work out (volunteering at "6-10 friends" should obviously be in the friends column), but the blank cells under the AMF donations have me puzzled.

The previous link to the survey results died, so I edited to update the link.

Do you have any sense of the extent to which people who put down 'friend' as how they got into effective altruism might have learned about EA via their friend taking them to a meet-up group, or might have ended up getting properly committed because they made friends with people through a meet-up group? I was just thinking about how I would classify myself as having got into EA through friends, but that you might think it was more accurate to describe it as a meet-up group in Oxford getting me involved.

I'm trying to make sense of all the missing data. It seems very strange to have such a high non response rate (nearly 20%) to simple demographic questions such as gender and student status, and this suggests a problem with the data.

You say here that a 'survey response' was generated each time somebody opened the survey, even if they answered no questions. Does that mean there wasn't a 'complete and submit' step? Was every partially completed survey considered a separate 'person'? If so, was there any way to determine if individuals were opening multiple times?

If each (however complete) opening of the survey was considered an entry (ie whatever data was entered has been counted as a person), that would suggest that individuals making several attempts to complete the survey are being multiply counted. That would be supported if the non response rates are generally higher later in the survey, which I can't tell from this report.

If multiple attempts can't be excluded, these numbers are unlikely to be valid. Missing data is a difficult problem, but my first thought is that a safer approach would be to only include complete responses

Thanks for the survey! An interesting read. One question, two comments:

1 How do I read the graph on p10?

2

"Reading the “The Four Focus Areas of Effective Altruism”, one would expect a roughly even split between (1) poverty, (2) metacharity, (3) far future / x­risk / AI, and (4) nonhuman animals. Above, instead of equal splits, poverty emerges as a clear leader \footnote: Statistically significant with a t­test, p < 0.0001." Though, maybe this isn’t fair. If we redefine meta­charity to also include rationality and cause prioritization, it takes the top slot (with 616 people advocating for at least one of the three). And if you take far future, x­risk, and AI as one cluster, it comes in third with 441 people advocating for at least one of the three. (Poverty, at 579, claims second place.)

It cannot be correct to say that poverty is clearly leading the four because in the next sentence you say that that metacharity was arguably more popular than it.

3

I don’t know of any public information on Giving What We Can members beyond the membership count and donation totals. They recently added a donation breakdown to their front page.

Thanks for all the work put into this, and it'll be good to see what thinking unfolds from it.