Making it easier for decision-makers to give feedback

By Catherine Low🔸 @ 2024-09-27T01:13 (+92)

I’m on the Community Health team at CEA, and am an advisor for EA Funds. My perspective here is informed by my experience in these roles, but I don’t speak for my colleagues - their experiences and preferences might be quite different. 

There’s been some discussion about the value of decision-makers (e.g. grantmakers, hiring managers, advisers) giving feedback to community members they interact with, and some discussion on the costs of giving feedback (I broadly agree with what Linch says in this post).

In my early days in EA I was desperate for decision-makers to give a whole lot more feedback – some of my applications were rejected and I didn’t understand why (at least not well enough to know what to do next). Many of my EA friends had similar experiences. I’ve now been involved in decision-making and advising roles for 4.5 years – initially in the Groups Team at CEA (where I would advise and sometimes decide on grants for group organisers), and more recently on the Community Health team (where I sometimes contribute to decisions e.g. who gets invited to speak or attend CEA events) and as an advisor for EA Funds.

Now I think the EA ecosystem would be better if decision-makers give a bit more feedback on the margin, but I’m not as bullish as I was before. 

I’m not going to go through costs and benefits of giving feedback here, I’m just going to share 

The idea is that some folks reading this who want more feedback in the community could make small changes that would increase the chance they/others get useful feedback**.

A related post that is worth reading is Ben Kuhn’s Tips for asking people for things. 

* It is not the feedback seeker’s job to make me happy of course! 

** I don’t know to what extent my preferences are similar to other people - results may vary! But seeing my preferences might nudge people to think about what others' preferences are. 

Useful things for people requesting/receiving feedback to know

1. There might have been a lot of factors that led to a decision. 

Sometimes there are multiple small reasons to be hesitant about a person doing a particular project – none of which are “deal breakers”. There is often so much uncertainty and decision-makers realistically have so little information such that the only option is to rely on small bits of information to update their view on the person/project. Each of these factors on their own might just be a small grain of sand on a scale, or be felt with low confidence, but together they might build up to tip the scale. 

In such cases, I find it can be difficult to give clear feedback that doesn’t result in the person overweighting one factor that went into the decision-making process.

2. In most cases feedback is for the combination of the people and the project.  I’m calling this the “people/project combo” in this post. 

So the best feedback is from people who have

a. An understanding of the field to assess the project

b. An understanding of the people involved to assess their fit for the project

When possible, seek feedback from people who have both! If someone doesn’t have an understanding of one (or any) of these, their feedback will have limitations so take into account the person’s knowledge of you and your subject area when working out how much to update on their feedback. 

This is why I sometimes get a little nervous when people ask a large number of people for feedback. E.g. emailing a proposal to a lot of people, or asking a lot of people during a conference, or, to a lesser extent, writing in a public post on the forum or on social media. This can result in the person getting a decent amount of feedback from people without b (and sometimes neither a nor b). [In addition, sometimes it takes a lot of community time]. I've seen wide requests for feedback that have resulted in (in my opinion) unhelpfully encouraging a not-great person/project combo, and also in a way that I think unhelpfully discouraged a strong person/project combo, because of the average lack of understanding of the field and the people. 

Feedback from a wide range of people can still be useful, so I don’t think people should stop asking for feedback in this way. My advice is to 

a. be conscious of the knowledge each person has about you and your field

b. maybe start with getting feedback from a smaller number of people, and then use that information to improve your plan and decide who to ask next. 

 

Things that have made me more happy to give feedback. 

  1. Clarifying the feedback needed: When people share clearly/honestly about why they are asking for feedback (and why they are asking me particularly), I know better how to help. 
    1. Is there a particular issue or worry they have that they want advice on?
    2. Do they want help with their project plan? (seems good if I have knowledge of the field)
    3. Or are they asking about whether I think they are a good fit (if I have knowledge of their skills/attributes)?
    4. Are they also wanting my help in some way (e.g. if I like their person/project combo they might be hoping I’ll connect them to other people in the community, or endorse their project in some way)? 
       
  2. Reasons for follow up questions: Sometimes people respond to my feedback and I am unsure whether they are: 
    1. Asking clarifying questions, as a result of my feedback being genuinely confusing 
    2. Sharing an appeal for reconsideration
    3. Or stating arguments against my point (with no particular request for reconsideration) 
      Knowing which of these it is will help me respond.
       
  3. Curiosity about perspectives on personal fit and project plans: I find I often have feedback to share when people ask more general questions about next steps for them/their project. E.g. “If you were in my shoes, what would you do?”, “do you think me undertaking this project would be net positive?” “do you think this rejection should update me significantly against doing <type of work>?”. Many of my thoughts will be low-confidence, but some people still express that it helps them. 
    Note: I might be unusual in this respect and other decision-makers might not like answering these general questions.

Things that make me more reluctant to share feedback

  1. When feedback is interpreted as coming from the organisation rather than the individual. In some cases people are speaking “on behalf” of their organisation, but other times they might be just sharing their own takes and it is interpreted as coming from their organisation. That is understandable - what a person says does of course reflect on their organisation to some extent. But each time I see that I feel just a little bit more pressure to say nothing unless it is important enough to discuss with my colleagues to check if they agree (which of course takes time and overall greatly reduces the amount of information I’m willing to share). 
  2. Rejecting the feedback: I appreciate it when people listen to and consider the feedback, and I’m disincentivised to give feedback when I don’t feel that is happening. This doesn’t mean you have to agree of course – but even if someone disagrees with the object-level feedback, the feedback likely still has some information value (it at least tells you the information “person X has the impression Y of the person/project combo”). 

  3. When I feel that people are requiring me to convince them fully of my point of view or decision (although I have messy thoughts on this). I don’t like this as much because

    1. sometimes it is time consuming, hard, or potentially counter-productive to convince others of a decision,
    2. it doesn’t acknowledge the reality of who is the decision-maker in this situation (this feels analogous to the post “The EA community does not own its donors' money”), 
    3. personal preference

    But I only weakly dislike this because our community is (and should be) a collaboration of people, holding each other to high standards - including (particularly) decision-makers. Explaining and justifying decisions is part of that. 

JamesN @ 2024-09-30T17:10 (+14)

Thank you for posting this, is it definitely nice to get a funders perspective on this!

From the other side (someone who has applied for grants and received little to no feedback on them), and having been involved in very large scale grant making through my governmental role, I fear your point (1) below is likely to be the greatest influence on grantmakers not providing feedback. Unfortunately, this I find (and did when I was a grantmaker and was prevented/unable to provide feedback) is often a cover for a lack of transparent and good reasoning practice in the grant decision process. 

The vast majority of EAs are aware of reasoning transparency and good Bayesian reasoning practices. I'd hope, as I assume many members of the EA community do, that EA grant makers have a defined method to record the grantmakers judgments and what is updating their view of a grants potential impact and likelihood of success. Not least because this would allow them to identify errors and any systematic biases that grantmakers may have, and thus improve as necessary. This should therefore be easily transferable into feedback to the grantee. 

The fact this isn't done raises questions for me. Are there such systematic processes? If not, how do grantmakers have confidence in their decision making a priori? If there are such processes to record reasoning, why can't they be summarised and provided for feedback?

The post you linked by Linch and the concern he raises that by being transparent about the reasons for not making a grant may risk applicants overupdating on the feedback seems unfounded/unevidenced. I also question how relevant given they weren't funded anyway, so why would you be concerned they'd over update? If you don't tell them they were a near miss and what changes may change your mind, then instead the risk is they either update randomly or the project is just completely canned - which feels worse for edge cases.
 

There might have been a lot of factors that led to a decision. 

Sometimes there are multiple small reasons to be hesitant about a person doing a particular project – none of which are “deal breakers”. There is often so much uncertainty and decision-makers realistically have so little information such that the only option is to rely on small bits of information to update their view on the person/project. Each of these factors on their own might just be a small grain of sand on a scale, or be felt with low confidence, but together they might build up to tip the scale. 

Catherine Low🔸 @ 2024-10-03T02:32 (+4)

Thanks for your questions James 

> This should therefore be easily transferable into feedback to the grantee. 

I think this is where we disagree - this written information often isn’t in a good shape to be shared with applicants and would need significant work before sharing. 

> The post you linked by Linch and the concern he raises that by being transparent about the reasons for not making a grant may risk applicants overupdating on the feedback seems unfounded/unevidenced. I also question how relevant given they weren't funded anyway, so why would you be concerned they'd over update? 

The concern here is that people can alter their plan based on the feedback with the hope it would mean that they’d have a better chance of getting the opportunity in the future. As Linch says in his post
> Often, to change someone’s plans enough, it requires careful attention and understanding, multiple followup calls, etc. 

I’ve personally seen cases where it seems that feedback sends a project off in a direction that isn’t especially good. This can happen when people have different ideas of what would be reasonable steps to take in response to the feedback.

But you’re right, Linch and I don’t provide evidence for the rate of problems caused by overupdating. This is a good nudge for me to think about how problematic this is overall, and whether I’m overreacting due to a few cases.

> If you don't tell them they were a near miss and what changes may change your mind, then instead the risk is they either update randomly or the project is just completely canned - which feels worse for edge cases.

I think it is most useful for decision makers to share feedback when a) it is a near miss, and b) the decision maker believes they can clearly describe something that the applicant can do that would make the person/project better and would likely lead to an approval. 

JamesN @ 2024-10-03T10:37 (+1)

Thank you for responding Catherine! It’s very much appreciated.

This should therefore be easily transferable into feedback to the grantee.

I think this is where we disagree - this written information often isn’t in a good shape to be shared with applicants and would need significant work before sharing.

I think this is my fundamental concern. Reasoning transparency and systematic processes to record grant maker’s judgments and show how they are updating their position should be intrinsic to how they are evaluating the applications. Otherwise they can’t have much confidence in the quality of their decisions or hope to learn from what judgment errors they make when determining which grants to fund (as they have no clear way to track back why they made a grant and whether or not that was a predictor for its success/failure).

SummaryBot @ 2024-09-27T20:07 (+1)

Executive summary: Decision-makers in EA should give more feedback, but there are challenges; the author provides advice for both feedback seekers and givers to improve the process.

Key points:

  1. Decisions often involve multiple small factors, making clear feedback difficult
  2. Feedback is most valuable from those who understand both the project and the people involved
  3. Feedback seekers should clarify their needs and reasons for asking specific individuals
  4. Decision-makers prefer when feedback is interpreted as personal opinion, not organizational stance
  5. Feedback givers are discouraged when recipients reject feedback or demand full convincing
  6. The author recommends balancing the need for decision-maker accountability with practical limitations on feedback

 

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.