Important EA-related questions EA would like to know from general public

By Daniel Hnyk @ 2019-12-13T20:55 (+14)

UPDATE: here is a document where I would like to concentrate all the ideas. Feel free to comment, suggest, edit. I will keep it updated based on feedback from both sides. I have time to send submissions until January 3.

UPDATE2: Some of the proposals have been accepted, more about the response here. I will update this post and comment in the case of some updates (e.g. how to get the data, but I don't think we'll know more sooner than in ~Q2 2020).

Hi.

I am working on a project where a company want to do a survey for about 6000 respondents in the UK & US about charities and they want to give this dataset to charities for free (they haven't done any public pledge or anything and they don't want to until they have started the collection at least, somewhat understandable). They know nothing about EA, but I am fairly sure they would be willing to incorporate questions related to it (be it "explicit" or not). The current version of the questionnaire has questions such as:

My question is if you are aware of any questions EA movement would be interested in asking in these countries (if any) or if you know who/where I should ask. It seems to me as a nice opportunity to cheaply gather some data. This survey is also likely to be repeated e.g. every quarter, so one could see trends over time. There are some limitations, like "easily answerable" (basically less abstract, multichoice) questions because of some other constraints and survey design, but I let's see what can be done.

The last thing is about any potential dangers with this initiative. Can this be counterproductive? Raise them as well, please.

I have time until December 23th, so not that much time left...

Thanks!


Sanjay @ 2019-12-14T14:48 (+10)

Great that you're doing this, thanks so much for raising this here!

Not sure if you're already aware of this, but you might want to be aware of other studies that have looked at similar questions. In particular the Money For Good study in the US and the equivalent in the UK were interesting, albeit somewhat dated. (I have the raw data for the UK study). You might want to have a look so that you can use consistent question wording whether the questions overlap.

Some questions that we at SoGive would be interested to know more about:

Some researchers are doing some interesting work on how people give -- the names Beth Breeze and Cat Walker spring to mind, although there are others as well. But you may well be too time-constrained to wade through all their work, in which case I suggest you just take a look at the Money for Good studies mentioned earlier.

More generally, very happy to discuss further. If you are willing to have a chat, let me know: sanjay [at] sogive.org

hnykda @ 2019-12-15T20:20 (+1)

Hi! Thanks a lot, these are all great!

For the donor population, I'd like to understand the split between what we at SoGive call Organisation-loyal, Cause-specific, and Open-minded donors (hopefully the labels are self-explanatory but if not please ask)

I believe so, but let me try.

  • Org-loyal: always give to a specific organization whatever it does
  • Cause-specific: throw money at a specific cause, e.g. global warming, doesn't necessarily care about the receiving org that much
  • Open-minded: happy to change their mind about both of the above

I am not confident I would be able to create high-quality survey questions though. Especially to distinguish between strong and moderate preferences. I have similar problems with the next bullet points. Maybe I am overthinking this though.

But you may well be too time-constrained to wade through all their work

Yeah, I have about a week to do send some suggestions as the study should be launched by Jan 2. Nevertheless, I believe that even if I managed to establish only a handful of high-quality questions, it would be good groundwork for future additions.

More generally, very happy to discuss further. If you are willing to have a chat, let me know: sanjay [at] sogive.org

Awesome, I would be really grateful for more help. Basically, they told me that if I put together some battery of questions in a doc, then they would incorporate it into the survey. My estimate is that they would put in at least 3 questions with 60% of probability, but I am having a really hard time to estimate this as there are a lot of factors in. But it seems to me that it's still worth to invest a few hours into this.

I created a document where I want to start summarizing this. Feel free to contribute directly.

David_Moss @ 2019-12-15T19:31 (+7)

I think questions about support for EA ideas in the general population would doubtless be interesting.

Unfortunately I think it is pretty difficult to ask questions about EA to the general public in an adequate manner. Since almost everyone is unfamiliar with EA ideas, statements of EA ideas are apt to be interpreted in line with more common folk ideas, rather than as expressing the EA ideas intended. For example, many statements of EA ideas ("We should only donate to the best effective charities" "We should do the most good we can do") can be interpreted completely platitudinously, so you find almost everyone agreeing with these statements even though almost no-one actually agrees with the ideas they are supposed to express. I think similar difficulties apply to asking whether people think those in the far future should be valued equally (see here and here)

Another specific problem is that almost no-one interprets "cost-effectiveness" correctly. I've run a number of studies examining how people think about thinking about cost-effectiveness in charitable decision-making, and I've found not only that most people naturally interpret "cost-effectiveness" to overhead ratios, but that even if you stipulate what cost-effectiveness means, and look at only those people who pass multiple comprehension checks putatively indicating correct understanding of the definition of cost-effectiveness, large percentages still cannot select which is the most "cost-effective charity" out of a pair of charities (A vs B) which save more lives with a given sum of money vs save fewer lives with the same sum of money but spend less on overhead costs.

I discuss this and some of the things I broadly think a good operationalization of EA should include here

That said, I'd be interested if you would ask people whether they agree or disagree with some statements along the lines of: "Some charitable causes are objectively better than others." "You can't compare whether different charitable causes are better or worse than each other."

hnykda @ 2019-12-15T20:37 (+2)

Yeah, these are my biggest concerns too, that's why I think it can't be done in that "straightforward" way...

That said, I'd be interested if you would ask people whether they agree or disagree with some statements along the lines of: "Some charitable causes are objectively better than others." "You can't compare whether different charitable causes are better or worse than each other."

This seems like a promising direction. I created a document where I am gonna try to somehow summarize these and turn into questions. Feel free to contribute directly. I would be super grateful.

MarisaJurczyk @ 2019-12-22T00:25 (+3)

I might be too late, but I was just cleaning through some notes of mine and found some questions related to this from various research agendas that I found interesting:

Some of these can be incorporated into your current questions (e.g. in the most important qualities when giving, you can mention low overhead). Maybe you can also have people choose or rank who they want to benefit from their giving the most (e.g. their community, animals, future generations, etc.).

Not sure if this is helpful, but glad that you're doing this regardless. :)

hnykda @ 2019-12-22T20:22 (+2)

Hi! Not too late! Thanks for ideas. The worry I have is about what @David_Moss mentioned above, e.g. people think that they _do_ donate to effective charities already (or that they do not donate to ineffective ones)... Do you have an idea about how to fight that?

Anyway, I am going to put these down into suggestions/considerations, it's good for inspiration at least.

david_reinstein @ 2019-12-23T09:41 (+2)

I may be too late to the game (been away and @DavidJanku only recently alerted me to this ) but some quick thoughts:

The current version seems to have many questions that will tell you about how people either 'consciously answer this question to themselves' or how they want to present themselves. It may not reveal their true motivations. There's a lot of work point in this direction.

I would try to focus more on very specific questions that permit less constructed justification and 'lying to oneself.'

It may be very helpful to present simple and yet that ask for a hypothetical response such as "which of the following charities would you be more likely to donate to?" and "how does the following information make you feel?" (although the latter may also offering for motivated reasoning). My recent paper with Robin Bergh some of this but with real donation choices; it still would be interesting to consider hypothetical choices and responses.

I have a wiki/hub that attempts to summarize much of the evidence on Charitable giving, with a particular focus on the consideration of effectiveness.

See INNOVATIONSINFUNDRAISING.ORG. There is also an underlying database I can share (with more detail and recent updates) if you message me at daaronr AT gmail.com

I have also done a lot of work recently summarizing the evidence on "How do people respond to effectiveness information".

E.g., pasting some text from a recent grant application:

So far, we have limited evidence on these questions, and the existing evidence is far from systematic or consistent Mixed results, e.g.,- Small et al vs Karlan, Parsons, and Reinstein et al work with Donor Voice; a Previous studies have largely relied on hypothetical and small-scale lab-based experiments(Metzger & Günther, 2015, Berman et al., 2018, Verkaik, 2016). Only a few large scale natural field experiments have been run. Karlan and Wood (2017) simultaneously varied emotional and cost-effectiveness information, with the latter presented largely qualitatively and in a particular ‘scientific’ credentials frame. Parson (2007) presented accounting information (uninformative about per-dollar effectiveness). In contrast, our field experiment project aims at large sample sizes in real donation environment, testing a set of particularly relevant and practical framings of real per-dollar impact information in the presence/absence of an emotional appeal (further measuring interaction effects, as in Bergh and Reinstein, 2019).

Please message me for more detail.

hnykda @ 2020-01-03T18:14 (+2)

Hi David. Thanks for the observation and a gotcha. I will pass this information over (and also note it down for myself, it's interesting :-) ). But I am skeptical in how much I have an influence around this. Nevertheless, I believe that the researches behind this survey will be interested as I mentioned in the post, there should be another window of possibility later on (as the study should be done periodically).

david_reinstein @ 2019-12-23T09:52 (+2)

I just added some links to the shared google doc also

hnykda @ 2020-01-03T18:09 (+1)

Post updated, see UPDATE2

Some of the proposals have been accepted, more about the response here.