Request for Guidance: Reaching Out to Charities Before Publishing Reviews and Our Concerns

By VettedCauses @ 2025-02-21T13:59 (+53)

Introduction

Vetted Causes writes reviews of charities. Thus far, we have published reviews of two charities. In both cases, we did not reach out to the charities before publishing reviews on them. We did consider doing so for our most recent review, but did not due to concerns which we've described below. 

We hope that through this post, members of the community can offer solutions that resolve our concerns. Should this happen, we would be more than happy to send reviews to charities before releasing them going forward. 

Acknowledging The Potential Benefits of Reaching Out to Charities Before Publishing Reviews on Them

We would like to start by acknowledging some of the potential benefits of reaching out to charities before publishing reviews on them: 

There are other potential benefits as well, which you can find stated and suggested by community members on our previous posts.

Potential Risks of Reaching Out to Charities Before Publishing Reviews About Them 

Note: none of the risks we mention are directed at any particular charity. When we begin evaluating any charity, we are concerned about all of these risks.

The two risks we are primarily concerned about relate to biased evidence, and unconscious biases.

Risk 1: Charities could alter, conceal, fabricate and/or destroy evidence to cover their tracks.

Much of the evidence we cite is from charities’ own webpages. Charities have the ability to change their webpages to potentially alter, conceal, and/or destroy evidence that we have cited. In some cases, there is also an incentive for charities to do this.

If we reach out to charities before publishing negative reviews about them, there is a risk that they will take these actions. While it is possible charities will still do this even when we don’t reach out before publishing negative reviews about them, it is riskier for charities to do this since the general public has already seen the review, and people may notice that evidence was changed after the review was released. This serves as deterrence against charities taking these actions. 

Currently, there are no known statistics on how often charities alter or remove evidence in response to critical reviews. However, there are many documented cases of evidence tampering in various fields. Additionally, police investigators often do not want an organization under investigation to know they are under investigation for fear they will tamper with evidence to avoid consequences. We believe similar concerns exist when we are investigating a charity and plan to release a negative review.

Prior to releasing our reviews, we have always taken screen recordings of important evidence. This was done in case one of the above scenarios were to ever happen. In the future, we also plan to create internet archives for the important evidence we use.  However, we remain concerned that in the case of a dispute, we would be accused of creating fake screen recordings/archives. 

If there is a third-party service that is trusted by the community that could verify the accuracy of our screen recordings/archives prior to us showing reviews to charities, we’d be much more open to the idea of showing reviews to charities before releasing them. Please let us know if you’re aware of one. 

Risk 2: Unconscious biases from interacting with charity staff.

When we evaluate a charity, we want to evaluate them based on their work, not based on how much we like their employees. Accordingly, we do not want to acquire unconscious biases. 

If anyone has solutions to this problem, please let us know below, as it would make us more open to showing reviews to charities before releasing them.  We would also like to acknowledge that we may be misunderstanding what people are suggesting when they say they'd like us to show our reviews to the charities before publishing them. If this simply entails sending them an email and nothing more, we are more open to that than having meetings with charity employees to discuss their review. 

Other Reasons 

Below are additional reasons we have been inclined to not reach out to charities before publishing reviews about them. This section largely reflects our own personal opinions.

Reason 1: Organizations should be held accountable for serious mistakes.

If a company makes a serious mistake that causes significant harm, even though it was a mistake, the company is generally still liable for the harm they caused. Simply correcting the mistake does not erase the harm that was caused. Similarly, if a charity makes misleading claims, it should not be allowed to quietly revise them without being held accountable for both the misinformation itself and the harm it caused.

If we reached out to a charity we have written a negative review about, we would hope they address the mistakes we identified. However, even if they addressed the mistakes, we believe they should still be held accountable for making the mistakes in the first place, and for whatever damage the mistakes caused. 

If we gave a charity a negative review before publishing it and they immediately fixed the problems we noted, we think it is likely that once we published our review, the charity would say something like “VettedCauses' review is about problems we have already addressed.” We believe this unreasonably reduces the level of accountability the charity is held to.

If anyone has a solution to this concern, please let us know below, as it would make us more open to showing reviews to charities before releasing them. 

Reason 2: Charities should be incentivized to provide sufficient and publicly stated evidence to justify their important publicly stated claims.

It is not acceptable for charities to make public and important claims (such as claims intended to convince people to donate), but not provide sufficient and publicly stated evidence that justifies their important claims. 

If a charity has done this, they should not be given the benefit of the doubt, because it is their own fault that there is not sufficient publicly stated evidence to justify their important claims; they had the opportunity to state this evidence but did not. Additionally, giving a charity the benefit of the doubt in this situation incentivizes not publicly stating evidence in situations where sufficient evidence does not exist, since the charity will simply be given the benefit of the doubt. 

Note: If a charity is unable to disclose all relevant evidence due to legal or strategic reasons, they should at least acknowledge this fact up front. A simple statement such as: "Unfortunately, we cannot disclose everything here due to legal reasons. Thus, the evidence we have presented is incomplete. We hope you trust that we have properly assessed our full evidence and reached an accurate conclusion" would be far more responsible than presenting the disclosed information as complete. 

Further, members of the public should not be expected to reach out to charities to figure out if they have hidden evidence that justifies their claims, since this also incentivizes not providing sufficient publicly stated evidence (charities could make claims based on poor evidence that is hidden, but many people would not know the evidence is poor since it is hidden). The expectation should be for charities to provide sufficient and publicly stated evidence to justify their publicly stated important claims. We are concerned that the norm of reaching out to charities before publishing critiques about them worsens these issues. 

We are interested to hear your thoughts about this concern. 

Conclusion

We apologize for violating the social norms of the community. It was not our intention to disrespect anyone. We are genuinely concerned about the issues above, and look forward to hearing the community’s thoughts and solutions.

Neel Nanda @ 2025-02-21T23:22 (+43)

I pretty strongly disagree with the case that you make care and think that you should obviously give charities a heads up and it is bad for everyone's interests if you don't. The crucial reason is that it is very easy to misunderstand things or be missing key context and you want to give them a chance to clarify the situation

Regarding your concerns, just use archive.org or archive.is to make archives or relevant web pages. This is a standard and widely used third-party service, and if the charity changes things secretly but the archived evidence remains, they look bad.

Regarding unconscious biases. I don't actually think this is a big deal, but if you are concerned it's a big deal then you can just email the charity and be like: Here is the document, we will publish it, but you have a chance to comment in this Google doc to point out factual inaccuracies and provide us evidence which we will attempt to take into account. And we will give you the opportunity to prepare a written response that can either be a comments immediately after this goes live or that we will include at the end of our piece. Readers can make up their own mind.

Regarding charities being held accountable for making mistakes, I don't think there's a big difference between the charity being publicly called out for a mistake and then they quickly fix it or the charity fixed it just before it was publicly called out. They still made the mistake and it's still obvious it was fixed because of you

Re your point about charities should be incentivised to provide sufficient public evidence. I think this is an extremely unreasonably high standard. Charities Should try to provide as much evidence as they can, but each person will have different objections and confusions and edge cases and it is just completely impractical to provide enough evidence to address all possible cases an adversarial reviewer might ask about. You can criticise charities for providing significantly insufficient rather than just mildly insufficient evidence. But ultimately there is always going to be the potential for misunderstanding and details that feel important to you that the charity did not predict would be important

VettedCauses @ 2025-02-21T23:58 (+1)

Re your point about charities should be incentivised to provide sufficient public evidence. I think this is an extremely unreasonably high standard.

Hi Neel, thanks for the reply. Could you clarify what standard you're referring to?

Neel Nanda @ 2025-02-22T11:21 (+6)

Quoting your post:

It is not acceptable for charities to make public and important claims (such as claims intended to convince people to donate), but not provide sufficient and publicly stated evidence that justifies their important claims.

If a charity has done this, they should not be given the benefit of the doubt, because it is their own fault that there is not sufficient publicly stated evidence to justify their important claims; they had the opportunity to state this evidence but did not. Additionally, giving a charity the benefit of the doubt in this situation incentivizes not publicly stating evidence in situations where sufficient evidence does not exist, since the charity will simply be given the benefit of the doubt.

Note that I interpret this standard as "provide sufficient evidence to support their claims in the eyes of any skeptical outside observer", as that's the role you're placing yourself in here

VettedCauses @ 2025-02-22T12:24 (+1)

Thanks for clarifying, Neel, and for giving us the opportunity to clarify our statements. Quoting our original post:

It is not acceptable for charities to make public and important claims (such as claims intended to convince people to donate), but not provide sufficient and publicly stated evidence that justifies their important claims.

By “sufficient evidence" we did not mean proof eliminating all possible doubt. We meant evidence that would satisfy a reasonable person. For example, a reasonable person thinks there is sufficient evidence that Neil Armstrong landed on the moon, but some people do not. 

Additionally, we do not expect charities to provide sufficient evidence for all of their publicly stated claims, just their important ones. For instance, if a charity claims to have provided 10,000 meals to homeless individuals, they should provide sufficient evidence that this actually happened. However, if they announce that their end-of-year party was at Disneyland, no additional evidence is necessary.

Jason @ 2025-02-22T15:15 (+17)

This rationale only works if you very carefully calibrate your claims. "Charity X has not provided sufficient, publicly available evidence to substantiate its claim to have provided 10K meals to homeless individuals" is a much different claim than "Charity X didn't feed 10K meals to homeless individuals." The former is about transparency; the latter is about whether the claimed results are true or not. An assertion that lack of transparency is not acceptable does not warrant an inference linking a lack of transparency to the falsity of the claim -- at least not without giving the charity a chance to correct its alleged transparency error ahead of time.

VettedCauses @ 2025-02-22T15:25 (+1)

Thanks for the insights Jason, we completely agree.

We are curious if you would you also agree that it is (1) reasonable to assert “Charity X didn't feed 10K meals to homeless individuals” and (2) unreasonable to give Charity X the benefit of the doubt, if:

  1. "Charity X has not provided sufficient, publicly available evidence to substantiate its claim to have provided 10K meals to homeless individuals" and
  2. There is publicly available and sufficient evidence indicating Charity X did not provide 10K meals to homeless individuals.
Jason @ 2025-02-22T17:13 (+10)

I think your second strand of evidence is doing almost all the work here. There's a big difference between evidence sufficient to establish a prima facie case, such that a reasonable mind would accept the conclusion if no additional evidence were offered and evidence that so conclusively establishes the truth that it's a waste of time to ask the charity for a response. 

As with other considerations, this is a fact-bound consideration. If you're relying on it to justify not giving a specific organization notice and an opportunity to prepare a response for concurrent publication, you can always say that -- and thereby accept the loss in your own credibility if readers later judge that the charity had a reasonable response that it could have provided if asked.

I would include situations in which the charity could establish the substantial truth of the assertion. E.g., if the charity could show that it made a math error and actually fed 9,823 individuals, that is information a reasonable reader would find highly relevant to evaluating the assertion of falsity. Not giving the charity a chance to concurrently explain that does a disservice to both the charity and the reader.

Neel Nanda @ 2025-02-22T18:57 (+6)

Yeah thi. In particular anytime you criticise an organisation and they are only able to respond a few weeks later, many readers will see your criticism but will not see the organization's response. This inherently will give a misleading impression, so you must be incredibly confident that there is no mitigating context that this organisation would give lest you do their reputation undue damage empirically. I think it is obviously the case empirically that when an organisation gets critiqued including your two previous examples that there is valuable additional context they're able to provide when given notice

VettedCauses @ 2025-02-22T19:03 (+1)

Thanks for the insights, Jason.

If you're relying on it to justify not giving a specific organization notice and an opportunity to prepare a response for concurrent publication, you can always say that

Could you clarify what you are saying we are relying on? 

Jason @ 2025-02-23T00:54 (+4)

It's a hypothetical about feeding homeless people, so it doesn't track any of your prior decisions not to give advance notice to the target organization.

In the hypothetical, you could assert that advance notice wasn't warranted because your external evidence was overwhelming enough that providing advance notice and opportunity for comment would have been a waste of time. This would be consistent with my interpretation of the community norm, which is that critics should provide advance notice unless they can (and do) articulate a good cause on a situation-specific basis to dispense with doing so. But if you did provide this justification, and the organization was later able to provide a response that materially changed the conclusions to be drawn, then your organization would suffer a loss in credibility as a result.

Neel Nanda @ 2025-02-22T20:59 (+2)

Seems pretty clear that he meant your second statement

There is publicly available and sufficient evidence indicating Charity X did not provide 10K meals to homeless individuals.

JamesÖz @ 2025-02-21T17:23 (+36)

IMO the risks you state are much less severe relative to missing key information about a specific charity (as likely happened with your Sinergia work) and therefore misleading people. This also makes people less likely to take your claims seriously in all future reviews.

Risk 2: Unconscious biases from interacting with charity staff.

When we evaluate a charity, we want to evaluate them based on their work, not based on how much we like their employees. Accordingly, we do not want to acquire unconscious biases. 

If anyone has solutions to this problem, please let us know below, as it would make us more open to showing reviews to charities before releasing them.  We would also like to acknowledge that we may be misunderstanding what people are suggesting when they say they'd like us to show our reviews to the charities before publishing them. If this simply entails sending them an email and nothing more, we are more open to that than having meetings with charity employees to discuss their review.

To clarify, yes I think most people think you should just share a Google Doc of your review and give the organisation time to leave comments about factual inaccuracies or other relevant context. I don't think anyone is suggesting you meet with organisations and discuss things via a call. If anything, it's best to share your review relatively early so you don't spend 100s of hours, as you claim you did with Sinergia, down some rabbit hole which may just be a lack of understanding or context-specific issues.

Lorenzo Buonanno🔸 @ 2025-02-21T20:41 (+4)

I strongly agree that the benefits of sharing the evaluation greatly outweigh the risks, but I'm not sure if sharing the it relatively early is best

  1. There is a risk of starting a draining back-and-forth which could block or massively delay publication. See e.g. Research Deprioritizing External Communication which was delayed by one year
  2. It would cost more time for the org to review a very early draft and point out mistakes that would be fixed anyway
  3. It could cause the org to take the evaluation less seriously, and be less likely to take action based on the feedback

 

I think the minimal version proposed by @Jason of just sending an advance copy a week or two in advance is an extremely low-cost policy that mitigates most of the risks and provides most of the benefits (but some limited back-and-forth would be ideal)

Aidan Alexander @ 2025-02-22T17:32 (+11)

From first hand experience, I think critics should make more of an effort to ensure that the charities have actually received your communications and had a chance to review it. When my organization was the subject of a critical post, the email from the critic had landed in my spam, so I only learned of the critique when reading it on the forum. 

What's more, as an organization of 2 people, one of which was on leave at the time of the notice, we couldn't have reasonably responded to it without stopping critical work to keep the lights on. The smaller the organization being critiqued is, the less flex capacity they have to respond quickly to these sorts of things, and so they should be given a longer grace period.

JamesÖz @ 2025-02-22T06:30 (+4)

Yep fair enough! That was one bit I wasn’t sure about and can definitely see the downsides of sharing too early. I guess the trade-off I was considering was Vetted Causes’ time spent on the evaluation but definitely think an advance finished version with two weeks notice would be something most groups would be happy with.

MichaelStJules @ 2025-02-21T16:43 (+19)

However, we remain concerned that in the case of a dispute, we would be accused of creating fake screen recordings/archives. 

If there is a third-party service that is trusted by the community that could verify the accuracy of our screen recordings/archives prior to us showing reviews to charities, we’d be much more open to the idea of showing reviews to charities before releasing them. Please let us know if you’re aware of one. 

https://web.archive.org/ seems good enough to me in most cases?

I think it doesn't do so well for Google Spreadsheets or videos, though:

  1. For example, this archive of ACE's CEA of ALI only includes the first tab of the sheet. I tried seeing if I could archive the other tabs via their separate links, but it just redirects me here, where only the one tab is archived. Maybe there's a way around this, or a better acrhiving service.
  2. AFAIK, videos are not archived at all, either.

I've also used https://archive.ph/. There's a browser plugin Archive Page for it, but I often get nginx errors when I try to archive pages with it. You can archive specific tabs by the tab links, e.g. here, but then it doesn't let you scroll through the sheet, which means you won't be able to access sheet cells you'd have to scroll to see. It also doesn't show how cells are calculated.

You could just save the whole sheet and upload it somewhere with a timestamp, and save archives anyway. Maybe there are better options than web.archive.org and archive.ph.

VettedCauses @ 2025-02-21T16:57 (+3)

Hi Michael, thanks for the reply. 

https://web.archive.org/ seems good enough to me in most cases?

That archive service is  great, we use them all the time. From our understand though, it is actually possible to manipulate web archives. 

MichaelStJules @ 2025-02-21T19:25 (+6)

Didn't they already address this specific vulnerability with the measures described on that page?

VettedCauses @ 2025-02-21T19:47 (+3)

The aforementioned page states that they took action "to mitigate these attacks," so from our understanding it is still possible to do. 

Also, the organization who completed the study still cautions users who rely on Wayback Machine (the archive platform that was manipulated).[1]

  1. ^

    https://rewritinghistory.cs.washington.edu/index.html See section "I rely on Wayback Machine -- what should I do?"

Lorenzo Buonanno🔸 @ 2025-02-21T20:19 (+4)

The original information is still archived, my understanding is that those attacks just inject other data that changes what is shown to the user, but as they mention it's easily detectable and the original information can still be recovered.

A bigger risk would be that the organization asks the archive to delete their data, but that would look very suspicious, and you could use multiple archives (e.g. https://archive.is/ )

VettedCauses @ 2025-02-21T23:14 (+1)

Thank you for your reply and technical insights, Lorenzo.


To clarify, we are actually not that concerned about archived documents being manipulated. From what we understand, this is extremely rare. 

What we are quite concerned about is that we will be falsely accused of manipulating archives, and the charity accusing us will be given the benefit of the doubt. They could cite articles like the one we cited earlier, and most people do not have the technical expertise to evaluate disputes over archive integrity.

Lorenzo Buonanno🔸 @ 2025-02-21T23:26 (+6)

I think that is extremely unlikely, they have a lot to lose as soon as it's confirmed that the archived data is not manipulated.

Also, from the page you cite:

we emphasize that these attacks can in most cases be launched only by the owners of particular domains.

So they would need to claim that you took control of a relevant domain as well.

But even if something like that happened, you could show that the archive has not been tampered (e.g. by linking the exact resource containing the information, or mentioning the "about this capture" tool that was added by the web archive to mitigate this)

VettedCauses @ 2025-02-21T23:54 (+1)

Thanks for the tool! It seems very useful. 

they have a lot to lose as soon as it's confirmed that the archived data is not manipulated.

We think our team still has some disagreements with you over how effective disinformation campaigns can be (especially when the disinformation is technical and the audience is mostly non-technical). That being said, we really appreciate your insights—you’ve made some great points.

Neel Nanda @ 2025-02-22T11:26 (+6)

I think the typical member of the EA community has more than enough technical skill to understand evidence that a web page has been edited to be different from an archived page, if pointed to both copies from a reliable source

Jason @ 2025-02-22T00:59 (+18)

Risk 1: Charities could alter, conceal, fabricate and/or destroy evidence to cover their tracks.

While this is possible in the abstract, I don't think it is generally applicable enough to credit without a case-specific application to the facts at hand.

For example: Your most recent post asserts that Singeria has made "false claims" and that this is an "important issue because Sinergia receives millions of dollars in grants from EA organizations." You note specifically that it has been a ACE top charity since 2018 and recently received a $3.3MM grant from Open Phil. I'll allude to that at points in the discussion as an example of what types of reports you might be producing, but emphasize that the discussion is entirely hypothetical.

One problem is that your argument for importance also underscores how difficult and risky for an organization to "alter, fabricate, and/or destroy evidence" in the way you describe. If the charity challenges your work, presumably you'd produce the screenshots. At that point, the charity would have to double down and accuse you of fabricating the screenshots; there's no plausible way to claim "misunderstanding" at that point in the game. 

That would be a serious charge, not least because I would expect (and generally hope) that grantmakers would impose an organizational death penalty on an organization that fraudulently claimed that an evaluator's screenshots were forged. Stuff like misattributed credit and poor calculations can be written off as non-malicious error, but deceit about something as concrete as what the organization's website said would be very hard to spin as anything other than career/organization-ending lack of trustworthiness.

In addition to general Internet archives, there are a number of places not under the organization's control where corroborating evidence could potentially be found. And I think it would be very hard for an organization to reduce the risk of information leakage to a level it deemed an acceptable risk:

VettedCauses @ 2025-02-22T12:50 (+1)

Thanks for your reply Jason—great to hear your perspective.

One problem is that your argument for importance also underscores how difficult and risky for an organization to "alter, fabricate, and/or destroy evidence" in the way you describe. 

We understand it is risky, but could you clarify why it would be difficult for an organization to "alter, fabricate, and/or destroy evidence?" In many cases, wouldn't it be as simple as changing values in a spreadsheet? 

Jason @ 2025-02-22T13:52 (+8)

It's not difficult to try. I believe it would be difficult to succeed (or even not get easily caught) because the evidence exists in too many third-party hands outside the organization's custody and control. Thus, absent special circumstances, it is very unlikely an organization would even try; the difficulty of success would deter them.

(I meant "risk" to refer more to the consequences of being caught -- that it would pose a grave risk to the organization's continued existence, and to the careers of those who attempted the tampering.)

John Salter @ 2025-02-21T22:13 (+18)

Updating your website feels extremely low urgency as a charity founder. Often you're facing existential threats, deadlines for funding applications, operational issues that can literally be the difference between life and death, and a million other pressing issues you know you'll never get round to because of the aforementioned.

I'm firmly on the side of sharing a draft.

Vasco Grilo🔸 @ 2025-02-21T15:59 (+13)

Thanks for asking for feedback, and for all your reviews! I very much encorage you to reach out to organisations before you publish your reviews. However, I also think reviews of organisations which have not been reviewed by them are much better than no reviews.

Risk 1: Charities could alter, conceal, fabricate and/or destroy evidence to cover their tracks.

I do not recall this having happened with organisations aligned with effective altruism.

 Prior to releasing our reviews, we have always taken screen recordings of important evidence. This was done in case one of the above scenarios were to ever happen. In the future, we also plan to create internet archives for the important evidence we use.  However, we remain concerned that in the case of a dispute, we would be accused of creating fake screen recordings/archives. 

I would be surprised if organisations aligned with effective altruism claimed the above type of evidence had been fabricated by you.

Risk 2: Unconscious biases from interacting with charity staff.

You can try correcting for this by not updating as much towards being positive about the organisation as naively warranted by the feedback you received from people working there. In addition, you can avoid meeting with people from the organisation, restricting yourself to sharing a doc with your review, and discussing it via messages.

Reason 1: Organizations should be held accountable for serious mistakes.

Organisations would be held accountable if risk 1 was mitigated.

Reason 2: Charities should be incentivized to provide sufficient and publicly stated evidence to justify their important publicly stated claims.

I agree with the points you make in this section, although I think sharing the reviews is still worth it overall.

Habryka @ 2025-02-21T18:11 (+15)

> Risk 1: Charities could alter, conceal, fabricate and/or destroy evidence to cover their tracks.

I do not recall this having happened with organisations aligned with effective altruism.

(FWIW, it happened with Leverage Research at multiple points in time, with active effort to remove various pieces of evidence from all available web archives. My best guess is it also happened with early CEA while I worked there, because many Leverage members worked at CEA at the time and they considered this relatively common practice. My best guess is you can find many other instances.)

Sarah Levin @ 2025-02-22T01:28 (+13)

At one point CEA released a doctored EAG photo with a "Leverage Research" sign edited to be bizarrely blank. (Archive page with doctored photo, original photo.) I assume this was an effort to bury their Leverage association after the fact.

Jeff Kaufman 🔸 @ 2025-02-22T16:05 (+4)

To the extent that you update against an org, of currently existing orgs this would be 80k, not CEA. At the time that this happened current CEA and current 80k were both independently managed efforts under the umbrella organization then known as CEA and now known as EV (more).

Separately, I agree this editing was bad, but doing it in the context of a review would be much worse.

Vasco Grilo🔸 @ 2025-02-22T09:24 (+2)

Thanks, Sarah! I have checked the links, and I agree that is a clear example of faking evidence. I assume this is an exception, and encourage CEA to disclose any similar instance where they have clearly faked evidence.

MichaelStJules @ 2025-02-21T19:29 (+4)

Were they successful in getting evidence removed from web archives?

Habryka @ 2025-02-21T20:24 (+7)

Yes, many of my links over the years broke, and I haven't been able to get any working copy.

Jeff Kaufman 🔸 @ 2025-02-22T16:06 (+6)

That sort of "it's hard to archive things reliably long-term" seems less relevant in the context of a review, where there's a pretty short time between sharing the doc with the charity and making the review public.

Vasco Grilo🔸 @ 2025-02-22T09:32 (+2)

Thanks for sharing, Habryka! If VettedCauses reviewed a random organisation recommended by Animal Charity Evaluators, and shared their review before publication, I guess there would only be a 20 % chance they would regret having shared the review specifically due to risk 1. What would be your guess?

VettedCauses @ 2025-02-21T16:24 (+3)

Thanks for the feedback, Vasco!

Risk 2: Unconscious biases from interacting with charity staff.

You can try correcting for this by not updating as much towards being positive about the organisation as naively warranted by the feedback you received from people working there. In addition, you can avoid meeting with people from the organisation, restricting yourself to sharing a doc with your review, and discussing it via messages.

These are fair points. We agree the risk of bias is likely minimal if this is how you handle it. 

Vasco Grilo🔸 @ 2025-02-22T17:43 (+4)

Thanks!

We agree the risk of bias is likely minimal if this is how you handle it. 

I think there is still some significant risk. My reviews of organisations very often become more positive towards the organisations after I integrate their feedback, although I try to update the reviews based on what makes sense to me instead of what appeals to the organisations. I try to counter the selection bias naturally present in the feedback I receive from organisations by actively searching for evidence against them after receiving their feedback, especially if I updated significantly as a result of their feedback. I search for contrary evidence not only in the updated parts of the review, but also elsewhere. In addition, you can ask feedback from people who you think are pessimistic about the organisations or their interventions.

Matt_Sharp @ 2025-02-21T16:45 (+5)

"Much of the evidence we cite is from charities’ own webpages. Charities have the ability to change their webpages to potentially alter, conceal, and/or destroy evidence that we have cited"

 

https://web.archive.org/ (aka Wayback Machine) regularly saves old versions of webpages. Maybe select a bunch of charities at random to see how thorough it is? In theory the charities could ask for old versions of their websites to be deleted from the archive, but there's no guarantees the archive would comply with their request and if they did it would look very suspicious if basically every other charity is on there.