Idea: Job Applicants Ask Organisations for their M&E

By GeorgeBridgwater @ 2022-07-08T14:11 (+29)

When organisations apply for grants, or talk with EA donors trying to acquire funds, they must attempt to provide evidence of their impact and theory of change; either explicitly, or through ongoing annual reports summarising the organisation's key outcomes and how these are connected to a positive impact on the world. This helps to keep organisations accountable, and reduces the risk of donors wasting their money on non-impactful interventions. 

Just like donors, applicants for jobs at nonprofits are offering significant amounts of their resources to the charity they are applying to. They are taking a potentially year-long bet that the organisation they are applying to is where they can have the most impact, and potentially accept a much lower salary for their labour than they could elsewhere. Given this investment, my impression is that a lot of job applicants don't put nearly enough due diligence into evaluating the organisations they are applying for. It seems that most rely on plausible stories of impact, and signals of EA alignment or acceptance from other EAs. I am guilty of this myself, with my first job applications to EA organisations relying on the general ‘EA affiliation’. 

Deferring to the soft judgement of the community makes sense in some circumstances, particularly for newer EAs. However, by the time you are experienced enough that you are applying for, and especially if you are being offered, EA jobs, you should not be trusting years of your career to simple signals of impact (with an expectation being something as strong as GiveWell recommendations).

In some ways, this little due diligence in job applications feels like a much more significant problem than donation decisions. A donor may split their donations among many potentially impactful organisations, and can easily pivot to other more impactful opportunities. A direct worker, however, who misallocates their work years could potentially have little or no impact for a large fraction of their career. An additional concern is that, the longer a worker has been aligned with an organisation, the more invested and biassed towards their current role they are likely to become. 

It seems that this imbalance in due diligence is driven by the typical implicit power dynamics when hiring or fundraising. Donors are well aware that the organisation needs their donation, and that there are many opportunities for them to donate elsewhere. On the other hand, job applicants may be unaware of how valuable they are to the organisation and the other opportunities the organisation has to hire. Even if they are aware of this, many job applicants have a very poor assessment of their competence and might not be aware of their counterfactuals. 

To remedy this, I would like to encourage job applicants to EA organisations to ask for a review of the organisation's internal monitoring and evaluation (M&E). This could be either in report form, or ideally through an introductory presentation to applicants who receive job offers, as one may do for potential donors. This report or presentation should walk applicants through the organisation’s theory of change, and the evidence that the group has accrued to date of their impact. 

Pros of asking to review an organisation’s M&E:

• Applicants can better evaluate the impact of the job they are offered, and therefore select the most impactful opportunities based on their own evaluation rather than deferring to signals of impact such as reporting EA alignment. 
• Organisations are provided with an external review of their M&E by potential experts in their field before they are significantly invested in the organisation. However, I expect a job offer would already be enough to bias their assessment somewhat.
• Acts as an additional method of holding organisations accountable to their stakeholders, and could encourage organisations with poor M&E to improve.

Cons of asking to review an organisation’s M&E:

• Applicants for many roles will have little or no experience with M&E and may not know how to interpret the information provided by the organisation. 
• Most of these applicants are likely to defer to the judgement of the organisation and thus won’t change anything. 
• There is also some risk applicants mistakenly update negatively against an organisation, for example, an early-stage start-up without sufficient proof of their impact yet. Although I should note that this is the current situation with many donors and the risk can be reduced through clear communication
• This may also lead applicants to update negatively against meta-organisations who tend to have weaker evidence of their impact when compared to direct organisations. However, this could equally occur the other way where applicants are swayed by weak fermi estimates that suggest a huge impact when the evidence to back it up is very weak.
• Time cost of sharing M&E reviews for organisations and applicants. 

Overall, despite the risk, I think asking for M&E reviews would be a good practice for EAs to follow when applying for roles, particularly those who have experience in M&E or research. The potential benefits to applicants and organisations’ accountability are likely to more than offset the potential problems, as seems to be the case in the funding landscape. 

I would be interested to hear if anyone has tried this in the past, and how potential employers would perceive such demands during the hiring process.