Giving What We Can - Pledge page trial (EA Market Testing)

By david_reinstein, JulianHazell, Giving What We Can @ 2022-05-16T22:39 (+56)

Introduction

GWWC has run an ambitious series of marketing and messaging trials. We aim to run more, including testing innovative features as part of our organizational update. We are targeting all parts of the "funnel", but focusing on outreach and 'conversion to taking a first meaningful step towards effective action'. [1]

The EA Market Testing team (EAMT) is working with GWWC (and others) to help design, implement, analyze, and preserve and communicate the results of these trials. We (EAMT) aim to help make the trials and innovations more directly successful, as well as increasing and sharing the insights and knowledge gained from these trials. 

You can see more of what the EAMT is up to in our (continuously updated) public gitbook here  (which includes the full report on the trial discussed in this post). [2] Below, we offer a summary of this trial, which we believe yielded meaningful results and insights. 

Note that the summary and the analysis (in the linked gitbook presentation) is still a work in progress, and we aim to update and improve it over the next months; feedback is very welcome!

Summary of trial and results

Giving What We Can (GWWC) has three giving pledge options, displayed in the 'Original presentation version' below.

Pledge page "Original"

From April-July 2021 they ran a trial presenting its 'pledge page' options in three slightly different ways. Considering 'clicks on any button' as the outcome:[3]

"Separate Bullets for Other Pledges":

 

These results may only apply narrowly to the GWWC pledge case, and even here, we have some caveats. However, it loosely suggests that, when making a call to action, it may be most effective to present the most well-known and expected option most prominently, and not to emphasize the range of choices. (See further discussion in the linked gitbook here.)

Getting people to take the GWWC pledge may be seen as an important outcome on its own. It may have a causal impact (see some descriptive evidence here) on getting people into, and engaged in the Effective Altruism community and in other effective altruistic activities, such as directly impactful career choices. E.g., over 20% of EA Survey respondents indicated that GWWC was important for them getting involved in EA.[4]

Once again, for more information about the trial, the statistical analysis, the results and the implications, we direct you to the gitbook section here.  We welcome comments below, in the gitbook[5], or in any other format.  

  1. ^

    We believe there is a strong case that engaging people in donations and pledges has an important direct impact, as well as being a significant entry point into other forms of EA involvement, such as career decisions.

  2. ^

     We also aim to provide our results, analysis, and data (where sharing is allowed) in a transparent web book, in a dynamic document format (using the Quarto tool). A very preliminary and work-in-progress version can be found here (watch this space). We are also looking for volunteers who can help with this statistical and data analysis and communication. I (David) am willing to provide mentoring is available for helping you learn these valuable open science/data science skills, and we also aim to find funds for paid work. Please contact me if you are interested, sharing your CV and a sample of your work, if possible.

  3. ^

    The differences below may not be statistically significant in standard frequentist tests; we have not done this analysis (yet).  In the Gitbook report HERE we present the Bayesian inference from Google Optimize, which generally suggests that the preponderance of evidence is in the direction described above. We aim to replicate and extend this analysis more transparently in the dynamic document.

  4. ^

    The share citing GWWC was even slightly higher for highly-engaged EA’s. Note that effective donations are an important route overall:  E.g., GiveWell was cited by nearly 35% of respondents.

  5. ^

    To comment in the Gitbook, please use the editor access link here and add comments (or suggest changes if you feel ambitious), or leave comments using hypothes.is and let us know.  

Rob Mitchell @ 2022-05-17T08:35 (+6)

Thanks everyone, this is very interesting and well worth having a look through the attached Gitbook.

Around the intuitive interpretation:

Perhaps giving people more options makes them indecisive. They may be particularly reluctant to choose a “relatively ambitious giving pledge” if a less ambitious option is highlighted.

It's possible that this is the reason, but there's an alternative interpretation based around the fact that GWWC is already quite well-known and referenced as 'the place you go to donate 10% of your income'. So if a lot of people are coming onto your page with that goal in mind, then it would make sense that the layouts that centre that option and make it as frictionless as possible will do better. Which is what we see here - the option centring a different option does much worse, but the one that does best is the one that most highlights the 10% pledge, not the one that contextualises it next to an even higher level pledge given equal space.

My own experience of using the site was very similar - I came on, looked around a bit for the 10% option I'd already decided on (in the original setup), then signed up. Things like favouring the middle option and the effects of anchoring are more relevant in a situation where someone has decided to buy, say, a broadband package but hasn't chosen which one; the lack of effect from them here might indicate that relatively fewer people are coming onto the page unsure how much to give.

You could try testing the 10% pledge next to the further pledge without the 1% pledge, but the really key thing feels like a post-pledge survey. 'Did you already know what you would pledge when you went on our website?' 'If so, did you consider giving at a different level when you saw the options?' etc. I'm sure you'd get a good response rate as people would be motivated to ensure others completed the pledge. Or if you already have this information, it would be really useful to see it!

david_reinstein @ 2022-05-17T18:12 (+2)

GWWC is already quite well-known and referenced as 'the place you go to donate 10% of your income'. So if a lot of people are coming onto your page with that goal in mind, then it would make sense that the layouts that centre that option and make it as frictionless as possible will do better

Thanks, that makes sense to me in a general senses.

I was thinking in this direction too but having a hard time putting in words 'why seeing options other than the expected one would make me less likely to follow through'.

Can you dive a little deeper into what the actual 'friction is' or 'what about seeing pledges other than the one I was planning to do would make me less likely to continue?' I guess my thought was that the mechanism would be indecision, need to take more time to think about this, or maybe a sort of 'hey, I am over-achieving here, do I really need to signal that I'm a 10-percenter when I could much more easily be a 1-percenter' ... but then I need to think about it more so I don't decide in the moment.

You could try testing the 10% pledge next to the further pledge without the 1% pledge,

This would be interesting, I agree.

Really key thing feels like a post-pledge survey. 'Did you already know what you would pledge when you went on our website?' 'If so, did you consider giving at a different level when you saw the options?' etc. I'm sure you'd get a good response rate as people would be motivated to ensure others completed the pledge. Or if you already have this information, it would be really useful to see it!

I think we would get some information from this. (AFAIK we don't have it but I could ask.) I'm not convinced it would be 'fully informative', because of the usual caveats about selection bias and people not always knowing/remembering what was in their minds. But still, it seems worth doing!

Rob Mitchell @ 2022-05-17T20:00 (+9)

'why seeing options other than the expected one would make me less likely to follow through'

I think the key is that 'following through' can mean several things that are similar from the perspective of GWWC but quite different from the perspective of the person pledging.

In my case I'd already been giving >10% for quite a while but thought it might be nice to formalise it. If I hadn't filled in the pledge it wouldn't have made any difference to my giving. So the value of the pledge to me was relatively low. If the website had been confusing or offputting I might have given up.

There are others who will already have decided to give 10% but haven't yet started. The pledge then would have a bit more value if there's a chance it could prevent backsliding but assuming the person had fully committed to giving at this level already, the GWWC pledge wouldn't be crucial to the potential pledger.

Finally, there are people who for whatever reason come across the website without yet having decided to give 10% (or even 1%) and make a decision to sign up when they're there. This is where the more standard marketing theory comes into play.

For the first two groups, the non-conversion is something like 'I can't even see what I'm meant to be signing up for. Never mind, it's not going to affect how I'll actually give anyway.' Friction in this case is anything that makes it harder to identify what the 10% pledge is and how to sign up to it.  I spent a couple of seconds looking between the three options but it was ultimately pretty easy to work out which one was the one I wanted. This would be even easier if it was the one main option.

For the third, it could well be 'There's too much choice, maybe I don't want to do it.' At any rate, it will be much different from people who had already committed to giving 10%.

The 'loss' to GWWC for all three looks the same but there's only a substantial loss to the wider world with the third group. 

I know people not always remembering what's in their minds can be an issue but I doubt it would be a problem on something like 'did you intend to give 10% when you arrived on the GWWC website?' and certainly not on 'have you already been giving 10%?' There's such a difference between the groups it would be really helpful to at least get an indication how they split out.

ThomasFowl @ 2022-05-19T07:43 (+1)

My pet theory about this is that original presentation looks very commercial and like most tech-subscription services, and that the simpler version looks, for lack of a better term, cheaper and also less commercial.