Refining improving institutional decision-making as a cause area: results from a scoping survey

By Vicky Clayton, dilhanperera, ibatra171 @ 2021-06-26T10:42 (+48)

Summary

• The improving institutional decision-making (IIDM) working group (now the Effective Institutions Project) ran a survey asking members of the EA community which topics they thought were in scope of “improving institutional decision-making” (IIDM) as a cause area. 74 individuals participated.
• The survey results will help target the Effective Institutions Project’s priorities and work products going forward. For example, the list of in-scope topics will form the guardrails for developing a directory of introductory resources to IIDM.
• Five out of the nine overarching fields we asked about were rated as ‘fully in scope’ by the median respondent. Leading amongst them was ‘whole-institution governance, leadership and culture’, with 77% of respondents rating it as fully in scope. Of the remaining fields, two were rated as ‘mostly in scope’ and two as ‘somewhat in scope’.
• ‘Determining goals and strategies to achieve them’ (e.g. ethics and global prioritisation research), had the smallest proportion of respondents rating it as at least somewhat in scope, although the number who did so was still a majority (62%).
• The respondents were demographically similar to the EA community as a whole but were more likely to work in government and policy and less likely to work in software engineering and machine learning / AI.

Why did we run a survey to scope out IIDM as a cause area?

As part of the IIDM working group (now the Effective Institutions Project), we ran a survey to help us refine the scope of improving institutional decision-making (IIDM) as a cause area (see the EA Forum post announcing the survey). The aim was to gauge the diversity of perspectives in the EA community on what “counts'' as IIDM. This helps us understand what the community thinks is important and has the most potential for impact. We hope that the results will shape the rest of our work as a working group and provide a helpful starting point for others as well. 

What did the survey involve?

We created a list of 67 topics based on input from a range of individuals involved in the IIDM community. The topics are mostly organised around academic disciplines and professional communities. Given that such a long list can be unwieldy, we then grouped these topics into overarching fields based on similarities such as level of the decision-maker, the focus of the decision etc. These fields are not ones that would be recognised as fields in academia but were useful in giving us broad brush insights into the types of topics which respondents may include as in scope.  

In the survey, we first asked the respondent whether they thought each field was within the scope of IIDM (see the screenshot below). 

Figure 1:

Respondents had the option to answer in more detail about each field. For each field for which they chose to do so, we asked whether they thought each of the topics listed under that field was within scope. As an example, one of the overarching fields was “Understanding and improving group and team dynamics in decision-making” and the topics within it were: improving group deliberation, improving voting (small scale e.g. in a committee), game theory, conflict resolution and procedural justice (see Figure 2). 

Figure 2

We provided respondents with a short description of each topic. The list of topics and their descriptions can be found here. We also asked for feedback on our working definition of IIDM and on what activities the working group should pursue.

We recruited respondents to the survey by posting the survey on the EA Forum, IIDM Facebook group and IIDM Slack workspace (link to join here) and also contacting individuals who had previously shown an interest in the cause area or who we thought would have strong (including sceptical) opinions about IIDM. We are incredibly grateful to the 74 people who took the time to fill in the survey. 

Results

Which overarching fields are considered in-scope for IIDM? 

The nine fields were:

• Understanding and improving whole-institution governance, leadership, and culture
• Understanding and improving group and team dynamics in decision-making
• Understanding how decisions (at all levels) are made and how they can be improved
• Using evidence and data to improve decisions
• Activities to support decision-making within government/international institutions
• Monitoring and improving the ongoing performance of institutions and their employees/collaborators
• Improving knowledge about the future to affect plans now
• Improving individual-level judgement and decision-making
• Determining goals and developing strategies to achieve them

As shown in Table 1 and Figure 3 below, five out of the nine fields were considered to be ‘fully in scope’ by the median respondent. Leading among them was ‘whole-institution governance, leadership and culture’, which was considered ‘fully in scope’ by 77% of respondents, and ‘mostly’ or ‘somewhat’ in scope by the others. The other fields rated as ‘fully in scope’ by the median respondents were considered to be at least somewhat in scope by the vast majority of other respondents.

Of the remaining fields, the median respondent rated two of the fields as ‘mostly in scope’ and the other two as ‘somewhat in scope’. Even among most of these lower-ranked fields, over 80% of respondents rated them as at least somewhat in scope. The one exception was  ‘determining goals and strategies to achieve them’ (e.g. ethics and global prioritisation research), where a smaller majority of respondents (62%) rated the field as at least somewhat in scope. This was also the field that most polarised respondents, with the most common responses being ‘mostly out of scope’ (27%) followed by ‘fully in scope’ (23%) and ‘mostly in scope’ (23%). Differences between the two distinct elements of this field - goals and strategies - might partly account for this: in the topic-level ratings for this field, ‘strategic planning’ was rated as at least somewhat in scope by almost all respondents (96%), but none of the other topics, which focused more on goals, attracted this rating from more than 6 in 10 respondents. 

Table 1: Fields by "in scope" rating

Figure 3

Which specific topics were considered in scope for IIDM?

Topics generally considered to be more in scope

The topics in Table 2 below were considered to be fully in scope by the median respondent. These topics also tended to have near unanimous agreement about being at least somewhat in scope (implementation science is somewhat of an exception, but only because it had a higher share of ‘not sure’ responses). A few other topics also had near unanimous agreement about being at least somewhat in scope, but were rated by the median respondent as being ‘mostly’ rather than fully in scope. These were: scenario planning, systems design / systems thinking; behavioral insights / science; strategic foresight; and scenario planning.

Table 2: Topics that the median respondent considered to be fully in scope

Topics generally considered to be less in scope

The topics in the table below to be only ‘somewhat’ in scope by the median respondent. None of the topics received a median rating of ‘fully out’ or ‘mostly out’ of scope.

Table 3: Topics that the median respondent considered to be somewhat in scope

Additional topics suggested by respondents

We also asked respondents to suggest additional topics that they thought were within the scope of IIDM but weren’t covered in the survey. The additional topics suggested were: social epistemology, practical long-termist decision-making, judicial decision-making and political theory (in particular the study of democracy). We did not then ask the respondents to indicate whether they thought the additional topics were in scope or not as we assumed that they would respond at least “somewhat in scope” given that they had suggested them. To proxy whether these suggested topics should be considered in scope, we’ve listed below the topics which we consider to be closest to these suggestions and the percentage of respondents who thought that those topics were at least somewhat in scope:

Other notable findings

The ratings provided by respondents who considered themselves to be working in an IIDM-related field (N=31) were broadly similar to those provided by those who said they were not or were not sure (N=36). In particular,  the top two-ranked fields and bottom-ranked fields (according to the % responding at least somewhat in scope) were identical for both groups). 

Nevertheless, there were two key differences between the two groups. First, those who said they worked in an IIDM-related field rated many fields as being more in scope than those who did not say they worked in an IIDM-related field. For example, 77% of these respondents said ‘determining goals and strategies to achieve’ them were at least somewhat in scope, compared to 50% of those who did not say they worked in an IIDM-related field. While we can’t say for sure what accounts for these differences, two possible explanations are: i) those who consider IIDM to be a high-priority cause area self-select into IIDM-related jobs; ii) professional experience with IIDM-related work increases beliefs about many of the fields we asked about being in scope.  

Second, the ranking of ‘decision-making support within government/international institutions’ was much higher among those who said they worked in an IIDM-related field (third ranked; 100% at least somewhat in scope) than among those who did not (second-last ranked; 81% at least somewhat in scope). 

Who took the survey?

67 out of 74 respondents (91%) shared information about themselves. Of these:

• Around 9 out of 10 respondents considered themselves to be a part of the EA community, with most being involved for 1-3 years (39%) or 4 or more years (42%). Of those who did not consider themselves to be a part of the EA community, 5 out of 6 considered themselves to be working in an IIDM-related field.
• Close to 1 in 2 considered themselves to be working in an IIDM-related field, while 34% did not and 19% were unsure.
• Around 9 out of 10 respondents were based in North America, Western Europe or Australia, with the remainder spread between Africa (1%), Eastern Europe (and former Soviet Bloc) (3%), Latin America and the Carribean (3%), and the Middle East, Asia and the Pacific (not including Australia) (4%)
• Around 2 out of 3 respondents were male, while 28% were female and 3% gave another answer.
• Around 2 out of 3 were aged between 24 and 34. Smaller shares of respondents were aged between 18-23 (17%), 35-49 (17%) or at least 50 (3%).

We also asked respondents about the areas in which they had at least 3 years of substantial work experience or graduate study with major focus, with the list of areas taken from the 2019 EA Survey. The most common areas of experience were government and policy (24% of respondents), management (22%) and economics (22%). Relative to the 2019 EA Survey, other overrepresented areas of experience included: movement building / public speaking / campaigning (16%), consulting (16%) and philosophy (13%), while software engineering was substantially underrepresented (9%, compared with 27% in the EA Survey). We didn’t have any respondents with a background in AI safety, machine learning, asset management, or personal assistance.

Limitations of the scoping survey

There were several limitations to the survey:

• Some respondents felt that the way they should answer the survey wasn’t made clear enough
• The breadth of the topics varied, meaning that some very broad topics may have been more likely to be considered at least somewhat in scope because counting entire academic disciplines e.g. political science as out of scope may have felt incautious; conversely broader topics may have also been less likely to be considered fully in scope
• The survey design, which was intended to reduce the time burden on respondents, meant that we need to be cautious when comparing the responses for the topics across fields due to a different sample answering for different fields
• Small sample size
• Respondents were broadly representative of the EA community in terms of the region they live in and their gender, but less representative in terms of professional expertise and age. We don’t know to what extent the respondents were representative of people doing IIDM work more generally.

Taking into account these considerations, the most robust results from both the perspective of a larger sample size and inter-field comparability are thus the results for the overarching fields as displayed in Figure 3 (rather than for the individual topics) although it is worth keeping in mind that these aren’t fields with established definitions outside of this survey. We have provided more detail on the limitations here.

How to define IIDM

Following the questions about whether the topics were in scope, we asked for feedback on our current working definition of IIDM:

“IIDM is about increasing the technical quality and effective altruism alignment of the most important decisions made by the world’s most important decision-making bodies. It emphasizes building sustained capacity to make high-quality and well-aligned decisions within powerful institutions themselves.”

A large majority of survey respondents who provided a response to the request for feedback on the definition (N=43) said that they disliked having “increasing … effective altruism alignment” within the definition. The most common concern expressed was that it may put off those outside of the EA community engaging with IIDM: “outsiders may be a bit suspicious if we say that we try to make institutions aligned with our ideology”. Several respondents went further, arguing that IIDM-like initiatives should be explicitly aligned with institutions' own preferences. (For organisations that originated within or work in close partnership with the EA community, those preferences would naturally align with the goals of effective altruism.) For others, it was more of a strategic consideration: “IIDM's comparative advantage is not in the EA-alignment part”.

Focusing on “the most important decisions made by the world’s most important decision-making bodies” was also seen by some respondents to be at the cost of considerations of neglectedness and tractability in addition to the importance of the decision.

We intend to take this feedback into account alongside other feedback we have received over the last 6 months as we refine the definitions we use both for IIDM in general as well as its component parts.

Prioritisation of activities

How did respondents prioritise activities to develop IIDM as a cause area?

We also asked respondents to score a range of potential activities the Effective Institutions Project could undertake to develop IIDM as a cause area, where a score of zero was “low priority” and five was “high priority”. As shown in Table 4 below, the only activity to receive the top score from the median respondent was “synthesise the existing evidence on ‘what works’ to improve institutional decision-making across academic fields”. Most of the other activities received a priority score of 4 out 5 from the median respondent and received a score of at least 3 out 5 from at least three-quarters of respondents. 

The two activities that received the lowest score from the median respondent were ‘compare IIDM to other cause areas’ and ‘help people start new organisations’ (2 out of 5). It is also notable that providing careers advice and funding recommendations (two common tools in an EA toolbox) also ranked lower down the priority lists, with a median score of 3 out 5. This may be because some of the higher-ranked activities are foundational prerequisites for these more practical outputs.

   Table 4: IIDM activities by priority rating

The prioritisation scores of people who said they worked in an IIDM-related field were similar to those who didn’t, particularly regarding the high priority of synthesising the existing evidence and the lower priority of the bottom ranked activities. However, there were some differences between the two groups on other activities. In particular, “Develop a theoretical framework for prioritising institutions or interventions within IIDM” was more likely to be scored highly by those who worked in an IIDM field (92% responding 3 out of 5, vs. 82%) while “Develop a research agenda for IIDM” was less likely to be scored highly by this group (79% vs. 89%). 

In our December blog post, we identified a number of key initiatives that we would focus on in 2021. We outline below what percent of survey respondents gave these  initiatives scores of at least 3 out of 5, along with the median score: 

• “Synthesise the existing evidence on “what works” in IIDM” (94%; median = 5/5)
• Community Engagement and Development: “Foster an IIDM community within the EA community” (89%; 4/5) , “Coordinate between actors within the EA community doing aligned work” (87%; 4/5) and “Build bridges between the EA community and people / organisations with aligned goals outside of EA” (85%; 4/5) and “Build bridges between the EA community and people working at or with important institutions” (81%; 4/5)
• “Developing a research agenda for IIDM” (84%; 4/5)
• “Developing a theoretical framework for prioritising institutions or interventions within IIDM” (84%; 4/5)
• [Resource directory: no direct equivalent activity included in the survey as this was a discrete project already underway]

These key initiatives overlap almost exactly  with those that received the highest scores from survey respondents . The only difference was the high ranking of  “Evaluate interventions to improve institutional decision-making” - not a key initiative identified in the December blog post - which was scored as a three or above by 85% of respondents, coming ahead of the research agenda the framework for prioritising institutions and some of the key community engagement activities.

Additional comments encouraged us to further prioritise: “Since this will be a relatively new cause area to be explored in the EA community I would be selective on what you can and want to achieve taking for instance capacity into consideration” and also to take into account what value we can add: “I think a lot of things are plausibly good ideas but it depends on the comparative advantage of the team.”

What’s next?

While this feedback from the community is not the only input into our prioritisation process , having a systematic way to consider which fields and topics are more in scope within IIDM allows us to make sure we’re building something which is reflective of what the community believes would be most impactful in relation to developing more effective institutions. Since a community of practice is defined partly by a common knowledge base, a list of more in scope topics gives us an idea of what core knowledge is important for activities ranging from educational programming to career development. As an immediate next step, we’ll be using the results from the scoping survey to inform the extent to which we focus on different fields within the resource directory. For example, we’re likely to provide in-depth resources for topics considered more in scope whilst we’ll provide only broad, high-level overviews of fields that are considered less in scope. The results of the survey may also shape how we prioritise our external engagement efforts, e.g. by focusing our attention on forging closer partnerships with organisations that concentrate on the more in scope topics. Finally, it gives us a starting point for where to look for interventions that may help improve institutional decision-making as part of the evidence synthesis.

With respect to our activities, on the whole, the survey results mostly seem to affirm our current plans for the group. We are currently considering what events and spaces are needed to foster an IIDM community within EA. We are also creating a process for a collaborative synthesis of knowledge about IIDM, which could act as a starting point for an academic synthetic review but is less formal (and so allows us to synthesise more and different types of knowledge). Both the resource directory and theoretical framework for prioritising institutions are currently underway. Whilst these will be somewhat living documents, the bulk of the work will be completed shortly. 

We have made one change in response to this feedback: given that none of the core team are academics and developing a formal research agenda scored poorly relative to the other key initiatives we’ve previously identified, we have parked this activity for the time being (although identifying unanswered key questions will likely be part of other work). On the other hand, even though it scored highly, we have decided not to prioritise evaluating interventions to improve institutional decision-making. Although there are undoubtedly many interventions that have not been formally evaluated in practical contexts, we believe that we get “greater bang for our buck” in synthesising the existing evidence and expect the synthesis to help us identify the gaps in knowledge and provide a roadmap for “field work” in the future.

Thank you!

Thank you so much to everyone who participated! If you’d like to chat to us about any of the above or getting involved in any of our activities, please contact us on improvinginstitutions@gmail.com 

About the authors

Dilhan is an advisor at the Behavioural Insights Team, focusing on designing evaluations and helping organisations make better use of data in decision-making. He analysed responses to most of the survey questions.

Ishita works at IDinsight. She brings experience working to support development sector leaders in using evidence in their decision-making. She developed the survey. 

Vicky is a data science manager at the What Works Centre for Children's Social Care, and uses her skills to improve the evidence base for social interventions. She proposed and coordinated the survey.

The Effective Institutions Project co-leaders Ian David Moss and Laura Green contributed to the list of topics and provided feedback on this post.

With thanks also to Aman Patel, Angela María Aristizábal, Marisa Jurczyk, Michael Noetel and Willem Roekens for contributing to the list of topics.

Appendix: Full topic-level results

Whole institutional governance, leadership and culture

Note: 100% of the overall sample reported that this field was at least somewhat in scope. The average proportion across the topics we listed in this field, from those who provided topic-level ratings, was 89%.

Group dynamics in decision-making:

Note: 100% of the overall sample reported that this field was at least somewhat in scope. The average proportion across the topics we listed in this field, from those who provided topic-level ratings, was 83%.

How decisions (at all levels) are made:

Note: 96% of the overall sample reported that this field was at least somewhat in scope. The average proportion across the topics we listed in this field, from those who provided topic-level ratings, was 91%.

Using evidence and data to improve decisions:

Note: 95% of the overall sample reported that this field was at least somewhat in scope. The average proportion across the topics we listed in this field, from those who provided topic-level ratings, was 84%.

Decision-making support within government and international institutions:

Note: 90% of the overall sample reported that this field was at least somewhat in scope. The average proportion across the topics we listed in this field, from those who provided topic-level ratings, was 84%.

Monitoring and improving ongoing performance of institutions:

Note: 90% of the overall sample reported that this field was at least somewhat in scope. The average proportion across the topics we listed in this field, from those who provided topic-level ratings, was 77%.

Understanding and improving interactions between institutions and across institutional ecosystems:

Note: Ratings were not collected for this field as a whole, from the overall sample, due to a mistake in survey implementation. The average proportion reporting at least somewhat in scope, across the topics we listed in this field, from those who provided topic-level ratings, was 80%.

Improving knowledge about the future to affect plans now:

Note: 86% of the overall sample reported that this field was at least somewhat in scope. The average proportion across the topics we listed in this field, from those who provided topic-level ratings, was 90%.

Individual-level judgement and decision-making:

Note: 84% of the overall sample reported that this field was at least somewhat in scope. The average proportion across the topics we listed in this field, from those who provided topic-level ratings, was 71%.

Determining goals and developing strategies to achieve them:

Note: 62% of the overall sample reported that this field was at least somewhat in scope. The average proportion across the topics we listed in this field, from those who provided topic-level ratings, was 64%.