Productive criticism: running a draft past the people you’re criticizing

By Lizka @ 2023-06-15T15:57 (+38)

I think running criticism past the people whose work is being criticized often helps make the criticism more productive, but it can be difficult. To make it easier, I'm sharing a step-by-step guide ⬇️ you can use. 

Please don’t feel like you have to read this whole guide or be super thorough if you’re thinking of running a draft past people. Don’t let perfect be the enemy of the good.

Outline of my suggestions on how to run criticism past people

  1. Make some key decisions about how you’ll share the criticism.
  2. Find an email address or reach out via the Forum.
    1. You can do this anonymously.
  3. Include relevant information — like a description of your timeline and boundaries on what you will and won’t be doing (like “I probably won’t have the capacity to respond to private replies before I publish the post”).
  4. Use a collaborative framing.
  5. If you want, start with one of the template emails that I’ve included.
  6. When you publish the post, consider including a note about the fact that you reached out in advance.

Why and when to do this

Some reasons to run criticism past people: 

  1. Avoid causing the people whose work you’re criticizing to scramble last-minute if they’d like to respond to your criticism in time for people to see their response. 
    1. A scramble like this might lead to wasted resources and increased stress.
    2. It can also lead to a delayed or missing response, which in turn could mean that readers end up with a worse understanding of the situation.
  2. Improve the criticism you’ve written, if you’re willing to modify your draft (and if the people you’re contacting are willing to share feedback or relevant information)
  3. Build a collaborative and criticism-welcoming culture in EA
  4. Build a good relationship between you and the people whose work you’re criticizing and a more collaborative framing for the criticism itself

I list more pros and cons in Appendix 1, where I also discuss that I think running a draft past people is especially useful when you’re criticizing work that isn’t public communication or research — like how an organization runs, specific decisions by individuals, etc. And for simplicity, I'm mostly focusing on criticism of work by people who are in the EA-adjacent network, although I think a lot of this could apply in other cases, too.

I outline my suggested norms on running criticism past people in Appendix 2.

Note: criticism can be incredibly useful

This post focuses on how we might mitigate the downsides of criticism, so I want to flag an assumption that I won’t emphasize much: criticism can be very helpful and we shouldn’t be discouraging it.[1] (I share some other assumptions in Appendix 3.)

Midjourney's productive criticism parrots — continued. This one is typing out a message to someone whose work they're criticizing.

How to run criticism past the people whose work you’re criticizing

Goals of the process & high-level overview

This guide is just a suggestion. You should do whatever makes sense for you (as long as it follows the Forum’s norms). In general, you probably want to:

  1. Achieve the benefits of running criticism past people
  2. While mitigating the costs — in particular, 
    1. Avoid delaying the publication of your criticism (including indefinitely)
    2. Avoid adding a lot of extra work for yourself
    3. Reduce other costs to yourself (stress, time, etc.)

These goals point to practical tips:

1. Make some key decisions

Will you share the criticism publicly?

I think that in some cases, criticism is better shared privately. I’m not discussing this much here; for this guide, let’s focus on criticism that will probably be public.

Are you running it past people?

My guess (and some others’) is that most criticisms would be more useful if they were run past the people whose work is being criticized first — unless there’s a specific reason to skip that. (More on pros/cons, and my suggested norms.)

 The rest of the guide will assume that you’ve decided to run criticism past the people being criticized.

Are you up for modifying your draft based on private responses from the people whose work you're criticizing?

I think it’s a personal choice depending on your situation, and I’m ultimately pretty unsure about which approach is better as a default. (And I think it's still very useful to run the criticism past people if you’re not willing to modify your draft; people may still appreciate seeing your criticism ahead of time.)

My suggestion: If you're unsure about what you want to do, communicate your uncertainty and give yourself a way out in case you start feeling like you’re getting sucked in. Consider capping the number of back-and-forth interactions you’re up for having ahead of time. (See this template email.)

2. Find a way to get in touch

You might already have contact information for relevant people. If you can’t easily find people’s email addresses, I’ve put some tips in this footnote.[2] 

You can get in touch anonymously

If you plan on using a pseudonym to share your criticism, you can make a pseudonymous Forum account and use that to message the relevant people. They won’t see the email address that you use. If they’re not on the Forum or you prefer email, you can use or make a pseudonymous email account (e.g. via ProtonMail, and here’s a relevant WikiHow article). 

3. Include relevant information

Most importantly, I think you should include:

  1. A clear timeline: when you will be sharing your draft more publicly, and if relevant, a deadline by which they can respond to your draft and expect that you will engage or update it (e.g. a week or two[3]):
    1. “I plan on posting this on [DATE], but you can let me know by [OTHER DATE] if you would like me to delay until [THIRD DATE].”
    2. “I will post this on the EA Forum on [DATE].”
    3. “I plan on posting this on [DATE]. If you think I should change something, it would be most useful to let me know by [OTHER DATE].”
  2. Whether you’re interested in changing your text or are just giving a heads-up, or something in between:
    1. “I don’t plan on updating this draft, but wanted to give you a heads up in case you wanted to pre-draft a response.”
    2. “I don’t expect to update this draft, but might if I learn relevant information by [DATE].”
    3. “If you notice claims you think might be wrong and you’re able to share relevant information by [DATE], I’d be happy to update the draft, but I don’t expect to have capacity for more than one back-and-forth. I may not see any emails after the first one that I respond to.”
    4. “I’d be excited to correct information that you think is wrong (I might add notes about why I misunderstood something, in that case), if you can respond by [DATE]. Let me know if more time here will help, and I’ll see if it seems reasonable to delay longer.” 
  3. What your draft is about

Other potentially-useful information to include:

  1. Context about why you’re writing this, or what you’re hoping to achieve
  2. Information that will help them know what kind of thing to give feedback on, If you’re up for updating your draft based on private responses
    1. How you formed your understanding of their work
    2. What your biggest uncertainties are
    3. What other feedback is useful
    4. How much you might update the draft

4. Use a collaborative & productive framing in your message

I’ve compiled some general tips for making criticism more collaborative and productive. I think an occasional failure mode is that people use a productive framing in the post itself, but reach out in a very blunt way. So try to consider your language in the email/message itself — flag something you appreciate about the targets of your criticism, focus on actions, etc.! 

5. Template/example emails

If you're using one of these templates, modify it based on what's true in your case.

Template Email 1: You don’t plan on responding or updating your criticism

Dear ____, 

I’ve written a draft [post/comment/…] that relates to your [work/organization/project/...]; you can see my draft [here — link]. I should say that I really appreciate [something you appreciate], but I focus on some criticisms I have, and my main point is ____. 

Please let me know if I should contact someone else about this. 

I plan on publishing this on the EA Forum on [DATE] (let me know if it would be noticeably easier for you if I published on [LATER DATE] instead). 

I don’t expect to update the draft much based on any responses here, as I won’t have the capacity for this, but wanted to give you a heads-up in case you wanted to draft a response that you can leave as a comment (no pressure either way from me) so that you won’t need to scramble when I’ve shared it publicly if you do want to respond. (If you’re interested in more context on why I’m reaching out in advance, see here.)

[If true (and consider making it more specific)] I appreciate a lot of your work, and what I see as your desire to help the world! I’m hoping that this helps us work on this project better. 

Best, 

[Your name or pseudonym]

Template Email 2: You might update the draft if they respond in a given timeframe

Dear ____, 

I’ve written a draft [post/comment/…] that relates to your [work/organization/project/...]; you can see my draft [here — link]. I should say that I really appreciate [something you appreciate], but I focus on some criticisms I have, and my main point is ____. 

Please let me know if I should contact someone else about this. 

I plan on publishing this on the EA Forum on [DATE] (let me know if it would be noticeably easier for you if I published on [LATER DATE] instead). 

I’d be interested in hearing if you think that this is incorrect in some way, or missing important context — I’d be happy to update this to be more accurate. If you can share information you think might help by [EARLIER(?) DATE], I’d be grateful! [If true] I don’t expect to have capacity for more than one back-and-forth, and I may not see any emails after the first one that I respond to. (If you’re interested in more context on why I’m reaching out in advance, see here.)

[If true (and consider making it more specific)] I appreciate a lot of your work, and what I see as your desire to help the world! I’m hoping that this helps us work on this project better. 

Best, 

[Your name or pseudonym]

6. When you publish the post, consider including a note about the fact that you reached out in advance

You could include one of the options below somewhere near the top, depending on your situation: 

Including a note like this helps share the norm of running criticism past people in advance; some people might see the note and realize that they might want to do something similar for future criticisms. It can also nudge readers to approach the post as a collaborative effort or a service that you’re providing for the people whose work you’re criticizing, which I think is usually a more helpful frame. 

  1. ^

    You can also read more about why I think criticism is important in the rationale we outlined for the Criticism Contest and in our companion resource.

    Here's a short (non-exhaustive) list of reasons to appreciate criticism: 

    - Criticism can help those criticized and others course-correct or form truer beliefs. If an important project is misguided in some way or could be improved somehow, that’s useful information!
    - Criticism can also help others learn and do better in the future, even if the specific project or work criticized doesn’t get corrected (e.g. because it's too late).
    - Welcoming criticism is a valuable cultural norm, and writing and appreciating individual examples of productive criticism can help sustain that norm.
    - Criticism can be a form of public good; it’s useful for the overall community and not usually officially someone’s job. 

    (The announcement about the winners of the Criticism Contest might also be relevant if you're interested in exploring this further.)

  2. ^

    The most reliable way to get in touch with a person or organization is probably to see if they have a website where they list an email address (some organizations’ websites may instead have a “Contact” form that you can use).

    If you’re not sure where to look, check the top bar on the website, and the footer at the very bottom. If you’re trying to get in touch with an individual and you don’t mind not being anonymous, you could also try reaching out on LinkedIn.

    Alternatively, you might be able to message them on the Forum. If they have an account, find their profile (you can search for it in the Forum search bar), then click on “Message” and write and send your message. This is a good idea if they’ve been active somewhat recently (you could check to see if they have decently recent comments). Most people will see a notification on the Forum and might get an email notifying them about this.

    You can also create group chats on the Forum’s messaging system by clicking on “Conversation options” and adding participants. Note that people’s usernames can be pretty similar, so I’d recommend checking that you have the right people once you’ve closed the “Conversation options” window — their usernames should appear under the conversation title, and you can click on the profiles there.

  3. ^

    It's hard for me to suggest a good default timeline here. 

    A commenter on a draft of this post listed some reasons to give more time than you might first think should be enough — I think these are easy to forget, so I'm sharing them here:
    - People are busy.
    - It might take a week for the message to reach the right person.
    - Stuff happens (e.g. relevant people are on leave or have personal emergencies).
    - In most cases, waiting one week or two/three weeks doesn't really make a difference for the criticism.
    - Responses might need to go through lawyers and/or PR approval, and this can take a while or be really costly.

    At the same time, I'm worried that setting expectations about slonger timeline can discourage criticism too much, or make things unnecessarily hard on people writing the post. 

    My tentative recommendation is to suggest a shorter timeline that you're happy with (e.g. a week), but proactively let the people you're contacting know that they can ask for more time if it would be helpful for them. (To avoid this dragging on if they ask for more time, you could give a concrete alternative timeline that's slightly longer but still has a specific deadline.)

Elizabeth @ 2023-06-17T18:17 (+15)

Can you clarify what your assumptions are about the criticized orgs? What would an org have to do before it became a bad idea to follow these steps?

It seems like these steps rest on an assumption of the org both wanting to be cooperative and having the skill to do it well. And not, e.g. explicitly promising confidentiality and then forwarding it to the exact person the author didn't want to see it. I believe that was an honest, human mistake, but that's exactly the problem: orgs are human- er, made up of humans- who have full plates and stuff gets missed. If organizations want the favor of previewing criticism, they need to take reasonable steps to make that easy and productive for critics.   

One particular way to do this would be to talk with people earlier in the process. The discussion I've seen around previewing criticism implicitly assumes there's a full polished blog post, ready to go except for the criticized org's response. That's a very annoying time to find entirely new facts, and imposes a very high tax on criticism. It also makes it easy for a defensive or merely human org to create a bunch more work for the critic without adding value. 

 If orgs made themselves available earlier in the process, to discuss concerns before someone invested tens of hours doing their own research and carefully writing it up, I would feel much better about strong norms of checking in with them ahead of time. Alas this probably necessitates spending a bunch of time with potential critics whose criticisms are stupid or who were never going to write them, which is pretty costly. But so is spending 30 hours writing a delicate, polished piece of criticism only to have the org create a bunch of work for you. 

Steven Byrnes @ 2023-06-15T23:13 (+13)

In this post the criticizer gave the criticizee an opportunity to reply in-line in the published post—in effect, the criticizee was offered the last word. I thought that was super classy, and I’m proud to have stolen that idea on two occasions (1,2).

If anyone’s interested, the relevant part of my email was:

…

You can leave google docs margin comments if you want, and:

  • If I’m just straight-up wrong about something, or putting words in your mouth, then I’ll just correct the text before publication.
  • If you are leave a google docs comment that’s more like a counter-argument, and I’m not immediately convinced, I’d probably copy what you wrote into an in-text reply box—just like the gray boxes here: [link] So you get to have the last word if you want, although I might still re-reply in the comments.
  • You can also / alternatively obviously leave comments on the published lesswrong post like normal.

If you would like to leave pre-publication feedback, but don’t expect to get around to it “soon” (say, the next 3 weeks), let me know and I’ll hold off publication.

(In the LW/EAF post editor, the inline reply-boxes are secretly just 1×1 tables.)

Another super classy move was I wrote a criticism post once, and the person I criticized retweeted it. (Without even dunking on it!) (The classy person here was Robin Hanson.) I’m proud to say that I’ve stolen that one too, although I guess not every time.

Lizka @ 2023-06-15T15:52 (+10)

Appendix 3. Other assumptions for the post

Following up on this note: below is a non-exhaustive list of other potentially-relevant assumptions that I won’t bother discussing but that I might rely on (to different degrees) in this post: 

  1. Criticism of someone’s work is more likely than other kinds of critical writing (like disagreement with someone’s written arguments) to be wrong or misleading because of an information asymmetry. It’s pretty common for criticism to be at least somewhat misleading (even if it still has significant and useful points). 
    1. When you’re writing about work that isn’t an entirely public output (communication/research that everyone can access), you’re more likely to simply lack context or be wrong about what you’re writing about. You can’t just reference specific parts of the work; the person or people who’ve done the work know things that you don’t. 
  2. It’s useful to help people orient towards criticism of their work in healthy and positive ways, which can mean trying to make the process less stressful for them. This isn’t against the critical spirit or the like.
  3. Wrong or somewhat misleading criticism of people’s work can be pretty harmful, especially if a response from those criticized doesn’t come right away.
    1. Readers come away with incorrect (and extremely negative) opinions of the work being criticized. 
      1. It can be harder for readers to tell for themselves whose side or claims they should believe than if this were a disagreement about public content (similar to the dynamics outlined in Assumption 1); they, too, lack information and will be potentially more susceptible to believing one side or the other based on who sounds more convincing or based on biases they already have.
    2. Readers could interpret the criticism as a broad and very negative judgement of all the work ever done by the relevant people, even if it’s not meant that way. 
    3. If a lot of readers start incorrectly believing that someone’s work is bad in certain ways, this can harm that person’s ability to do other work. 
    4. It’s really stressful when someone shares incorrect criticism of your work publicly, especially if you don’t feel like you have the chance to defend yourself in time for people to see your counterpoints. 
    5. Stress or negative experiences like this lead people to be overall more negative about criticism, even when it’s accurate, productive, kindly presented, etc. 
  4. Responses from the people criticized are worth showing alongside the criticism even if we think that the criticism is on point. In particular, it's useful to give them the chance to write them in time to post an early comment on your post.
    1. Showing responses alongside criticism lets readers form more independent opinions. 
    2. Showing responses like this helps make the criticism feel more collaborative.
  5. People writing criticism in EA are often in a collaborative relationship with the people whose work they’re criticizing; this isn’t a zero-sum relationship, they have common goals, they don’t want to cause unnecessary harm, etc..
  6. People who have less context on a project or who are missing significant information can still have incredibly useful perspectives and suggestions to contribute. 
    1. Maybe the work could be improved in a way you have special experience with, or you’ve spotted mistakes others missed — the people working on a project could be biased, and not tracking relevant information, etc. 
  7. Criticism should not be suppressed. 
  8. Indefinite or extended delays (of criticism and of other stuff) are pretty dangerous and can result in ~silent vetoes. They should be avoided.
  9. People doing work in EA are humans whose feelings matter and can be helped in productive ways. 

(There's almost certainly more.)

Lizka @ 2023-06-15T15:48 (+6)

Appendix 2. Where I personally think the norm should be on whether to run criticism past people

I think it’s generally good to give people a heads-up if you’re writing a post criticizing their work (and I tentatively think a week or two is a reasonable timeline). 

I don’t think we need to push people to be willing to update drafts if they get a private response to their draft before the criticism is posted publicly. I know of multiple cases where the critic got pulled into an extended back-and-forth as a result of doing something like this, and I suspect that this kind of thing sometimes causes people to just not post anything in cases where something should have been posted. (There are still clear benefits to being willing to update drafts, but there are also real costs.)

I also think that running criticism past the people whose work is being criticized should be a soft norm that you can override if your criticism is time-sensitive (e.g. there’s something happening tomorrow that you want to inform the Forum audience about) or if you have good reason to believe that people don’t share your fundamental values, are not acting in good faith, might pressure you to stop you from publishing, or would react in a hostile way for one reason or another. (E.g. if someone were to write a criticism about how MacDonalds shouldn’t be serving meat, I don’t think they should feel obliged to first get in touch with MacDonalds. Jeff notes another example in his “EDIT” here.)

In particular, from a moderation perspective: we will not ban people from the Forum for posting criticism without running it past anyone. 

Note: if you’re thinking of sharing very personal information or something closer to an unverified rumor, I’d more strongly push you to run it past people in advance. You can check with the moderation team if you’re not sure whether something is personal. See also the policies on revealing personal information on the Forum.

Lizka @ 2023-06-15T15:45 (+6)

Appendix 1.[1] Reasons to run criticism past the people whose work you’re criticizing, and reasons to not do it

Please feel free to suggest more!

You can see my overall take here. 

This post has a somewhat similar discussion. 

  1. ^

    These are pretty rough notes that I pulled out to make the main post easier to use as a guide. The comment format is somewhat awkward, but it might work — I'm curious to hear how people feel about it. 

Vasco Grilo @ 2023-06-23T07:17 (+4)

Thanks for the pretty useful tips, Lizka!

If you can’t easily find people’s email addresses

The email of the person can usually be inferred from the name, and domain of the website of the organisation. For example, to contact you (Lizka Vaintrob) who work at CEA (whose website domain is centreforeffectivealtruism.org), I would consider the following (in this order):

Instead of sending my email to all of the above, I would use Email ID Checker to find the 1st valid one. It looks like the 1st is valid:

Chris Leong @ 2023-06-15T16:28 (+2)

“ Are you up for modifying your draft based on private responses from the people whose work you're criticizing?”

If you’ve solicited feedback, I would suggest that your obligation to modify your draft depends on how strong their response is.

If it’s clear that your draft contains significant factual inaccuracies, then publishing reflects poorly on you (though you shouldn’t automatically feel a need to respond to attempts to dispute a fact, as often they may be pointing out something minor or you may have reason to doubt their account).

If the average reader after reading both your post and their response will come away with an impression that your critique was ideological or one-sided, then you should probably edit your draft, though there might be exceptions (say if someone is traumatised to the point where they lack the ability to be objective, so the alternative would be not publishing at all).

To be clear, I’m not saying that you need to edit just because they’ve made some good or reasonable points. I’m more suggesting that you shouldn’t ignore points if this would make a good faith reader feel like your article was pushing them away from the truth.