Are you working on a research agenda? A guide to increasing the impact of your research by involving decision-makers

By KarolinaSarek🔸 @ 2019-09-24T11:00 (+70)

Why is some research conducted with very high quality, yet it does not affect decisions on where to direct our time and money? What makes research relevant, important, and ultimately lead to positively impacting the world? A considerable portion of resources aimed at doing good at the world goes towards conducting research. In just the animal advocacy space (often considered one of the least research-heavy causes), about $9 million and 40 full-time researchers in the past five years went to work. That number is growing. With increasingly more organizations focusing on research, the importance of designing an effective research agenda is also growing. With so much attention on research, there is a high degree of importance on creating an impactful research agenda. In this post, I will present one meta-method for improving the impact of a research agenda. This post starts by explaining the importance of the theory of change for your research and then elaborates on a method to involve decision-makers in the process of creating your research agenda to maximize the impact of research.

What makes a good research agenda?

A good research agenda should be:

• Goal-directed and utilized—Research should arrive at a conclusion and have a clear path to impact through involving beneficiaries.
• Achievable—Research should be planned considering available resources and the tradeoff of depth for breadth.
• Concise—The agenda should be based on an exhaustive decision-making process and prioritization.

To achieve those qualities, we need to:

1. Build a Theory of Change. Understanding the context around your organization will clarify who benefits from your research and how your organization can have an impact. It will showcase expected outcomes over the short, medium, and long term. Very likely, the main way your research could have an effect will be through informing other decision-makers in your space: funders, direct organizations, other researchers, etc. By directly involving them in creating an agenda, you will ensure that your research will be used.
2. Put a significant portion of your total time into agenda creation. Broadly speaking, you should switch from agenda creation to direct research once the expected value of the marginal effort put into thinking about an agenda equals the expected value of research. This calculation deserves a separate post, but generally given that most organizations are creating an agenda for the next year, I would recommend spending at least 10% of your time on designing the research agenda.
3. Generate an exhaustive list of ideas before picking a few of them. One of the most effective ways to produce creative research ideas and, therefore, have a higher counterfactual impact is to generate a great deal of them and strongly prioritize them based on how they score on relevant metrics. For example, funds potentially influenced, number of hours of other organizations influenced .

In this post, I will cover the ideas for generating a research agenda that is goal-directed and utilized. There are many other principles that contribute to high-impact research. I plan to cover some of them during my talk at EAG London in 2019 and plausibly in a series of blog posts focused on maximizing expected value in research.

Build a Theory of Change

Theory of Change (ToC) is essentially a comprehensive description and illustration of how and why the desired change is expected to happen in a particular context. It shows how expected outcomes occur because of your work. Having a clear path from the original research to its endline impact will help you choose research questions that are most relevant to your goal. ToC is widely used in the charity world to help organizations and donors ensure the activities conducted directly relate to their endline goals. The same methodology can be used for research-focused organizations.

ToC can help an organization improve its strategy, measurement, communication, and partnership work and, consequently, increase the impact of your work. When you have a complete Theory of Change, it:

• Helps your team achieve a shared understanding of a project, its aims, and how it leads to an impact. Therefore, it is not only easier to stay on track toward your main goals, but also it helps your staff to feel involved and motivated, showing how their research contributes to a long-term impact.
• Helps to identify hidden assumptions and mitigate risks. A range of assumptions likely underlies the design of your project, some of which may be unfounded, out of date, or inconsistent with evidence. By challenging and clarifying them, you reduce the risk of one of the steps failing and making your project less impactful.
• Makes your organization more effective. Having a clear and testable hypothesis about how impacts will occur allows your organization to have a clear goal, identify where activities are not contributing to your goals, take action to correct course, and provide a blueprint for evaluation with measurable indicators of success identified.
• Serves as a powerful communication tool to capture the complexity of your initiative. Having a visual representation of the desired change, how to see it, and how you expect it to happen can quickly convey the project's aims to stakeholders, including funders and other research organizations with which you collaborate.
• It is an agreement among stakeholders about what defines success and what it takes to get there. Developing a theory of change in collaboration with other organizations can establish consistency around outcomes and define research questions that you must answer to help decision-makers redirect their resources to higher-impact areas.
  
  

Although all these benefits are significant, in this post, I will elaborate on the last benefit, as this is particularly relevant for research and provide a template to help you decide what research questions are most important to answer.

Crucial elements in your ToC

When analyzing your path to an impact, you need to think on a few levels. Those levels create the core of your theory of change and showcase a simplified model of your activities. For ToC to be fully informative, it should include more information, for example, assumptions about how a project will work, stakeholders, and how they relate to your intermediate outcomes.

Much has been written on how to create a theory of change, and explaining it is outside of the scope of this post. I recommend these three sources for a comprehensive guide to building your theory of change

Involve decision-makers to create effective research agendas

In most cases, your Theory of Change will involve your research informing other decision-makers in your space. For example, Charity Entrepreneurship’s (CE) research aims to influence ourselves (what charity we recommend starting), attendees of the CE incubation program (what charity they will start), entrepreneurs starting new EA charities independently (what charities they will start), EA funders (what new projects are worth funding), and other EA-minded organizations working in a given cause area (what intervention to implement). Thus, there are five groups of stakeholders in our research. Below, I present how to involve those stakeholders in the decision-making process.

1. Create a list of all stakeholders and decision-makers that will be informed by your research. (You can use your ToC for this step.)
2. Assign a weight to all of them according to their importance (amount of resources affected by them and odds they will be affected by your research).
3. Before showing them your list of potential research questions, ask them about questions, problems, and interventions that you could research that will help them with their decision making. Add those questions to your list.
4. Ask them to rate each research question on your list according to how important an answer to this question is for them, how much it could update their views, the effect of this update, etc.
5. Calculate the overall score for each question. Pick the most important one to answer.
6. Based on the overall score and availability of your resources, choose what questions you will explore.
7. Inform decision-makers what and when you will share your research with them.

The above process will ensure that you are making your research relevant to your short- and long-term goals and that your research will be more likely to be utilized and achieve higher impact. You may notice it is a cross-application of spreadsheet decision making (which you can see more on here).

Conclusion

One of the key elements of impactful research is its relevance to decision-makers in your field whose actions you may affect. To ensure that your research will be utilized, remember to root it in your Theory of Change and involve stakeholders to systematically choose the most important questions to explore.