AMA: Ian David Moss, strategy consultant to foundations and other institutions

I love the way you phrased this question -- in fact, one of the reasons why I'm such a big believer in theories of change (so much so that I wrote an introductory explainer about them) is that they are excellent for revealing strategic mistakes in a client's thinking.

A frequent pitfall I come across is that the originators of an organization or program often fall in love with the solution rather than the problem. By that I mean they see a problem, think immediately of a very detailed solution for that problem -- whether it's a software platform, some other kind of technology or innovation, an adaptation of an existing idea to a new audience or environment, etc. -- and get so invested in executing on that solution that it doesn't even occur to them to think about modifications or alternatives that might have higher potential. Alternatively, the solution can become so embedded in the organization's identity that people who join or lead it later on see the specific manifestation of the solution as the organization's reason to exist rather than the problem it was trying to solve or opportunity it was trying to take advantage of.

This often shows up when doing a theory of change for a program or organization years down the line after reality has caught up to the original vision -- day-to-day activities, carried out by employees or successors and shaped through repeated concessions to convenience or other stakeholders, often imply a very different set of goals than are stated in the mission or vision statement! For that reason, when doing a theory of change, I try to encourage clients to map backwards from their goals or the impact they want to create and forget for a moment about the programs that currently exist, to encourage them to see a whole universe of potential solutions and think critically about why they are anchored on one in particular.

Hi Michael, there are some sample project descriptions over at my website, but I'll paste a couple here for convenience: 

For more than 18 months, I worked with Democracy Fund’s Strategy, Impact and Learning team to bolster organizational capacity for strategic decision-making and develop a framework for risk assessment and mitigation across the organization. Deliverables included a training for 35+ senior and program staff covering forecasting skills and decision analysis, a concept paper, and recommendations to strengthen the approval process for nearly $40M in annual grantmaking. (2018-20)

I advised the Omidyar Network on the development of its Learning & Impact plan, with particular attention to designing team and organization-wide accountability systems that incentivize smart decision-making habits and practices as an alternative to traditional outcomes-based accountability. In addition, this engagement helped support the creation of a framework to help philanthropic institutions respond to the uncertainty created by the COVID crisis. (2020)

In partnership with BYP Group, I developed theories of change for two grant programs administered by Melbourne, Australia-based Creative Victoria (a state government agency). In addition, I worked directly with nine Creative Victoria grantees to create evaluation frameworks for their funded projects, which sought to use creative industry assets to accelerate progress on longstanding social issues such as mental health, social cohesion, and gender equality. (2018)

Those should give you a high-level sense of what I do, but I'm happy to answer more specific questions as bandwidth allows.

1. Be aware of your decisions in the first place! It's really easy to get so caught up in our natural habits of decision-making that we forget that anything out of the ordinary is happening. Try to set up tripwires in your team meetings, Slack chats, and other everyday venues for communication to flag when an important fork in the road is before you and resist the natural pressure people will feel to get to resolution immediately. Then commit to a clear process of framing the choice, gathering information, considering alternatives, and choosing a path forward.
2. Match the level of information-gathering and analysis you give a decision to its stakes. Often organizations have rote processes set up for analysis that aren't actually connected to anything worth worrying about, while much more consequential decisions are made in a single meeting or in a memo from the CEO. Try to establish a discipline of asking how much of your/your team's time it's worth spending on getting a decision right. Try to ensure that every piece of knowledge your team collects has at least one clear, easily foreseen use case in a decision-making context, and dump any that are just taking up space.
3. Try to structure decisions for flexibility and option value. Look for ways to run experiments and give yourself an out if they don't work, ways to condition a decision on some other event or decision so that you aren't backed into making a choice before you have to, ways to hedge against multiple scenarios. Obviously, there will be situations when there is one correct choice and you need to go all-in on that choice. But in my experience those are pretty rare, and clients are more likely to make the opposite error of overcommitting to decision sequences that overly narrow the set of reasonable future options and cause problems down the line because of that.

Great questions!

1. I'm on record as believing that working on EA-style optimization within causes, even ones that don't rise to the top of the most important causes to work on, is EA work that should be recognized as such and welcomed into the community. I got a lot of pushback when I published that post over four years ago, although I've since seen a number of people make similar arguments. I think EA conventional wisdom sometimes sets up a rather unrealistic, black-and-white understanding of why other people engage in altruistic acts: it's either 100% altruistic, in which case it goes into the EA bucket and you should try to optimize it, or it's not altruistic at all, which case it's out of scope for us and we don't need to talk about it. In reality, I think many people pursue both donations and careers out of a combination of altruistic and selfish factors, and finding ways to engage productively about increasing the impact of the altruism while respecting the boundaries put in place by self-interest is a relatively unexplored frontier for this community that has the potential to be very, very productive.
2. This depends on whether you center your perspective on the EA community or not. There are lots of folks out there in the wider world trying to improve the functioning of institutions, but most of them aren't making any explicit attempt to prioritize among them beyond whether they are primarily mission- or profit-driven. In this respect, the EA community's drive to prioritize IIDM work based on opportunity to improve the world is quite novel and even a bit radical. On the EA side of things, however, I think there's not enough recognition of the value that comes from engaging with fellow travelers who have been doing this kind of work for a lot longer, just without the prioritization that EA brings to the table. IIDM is an incredibly interdisciplinary field, and one of the failure modes that I see a lot is that good ideas gain traction within a short period of time among some subset of the professional universe, and then get more or less confined to that subset over time. I think EA's version of IIDM is in danger of meeting the same fate if we don't very aggressively try to bridge across sectoral, country, and disciplinary boundaries where people are using different language to talk about/try to do the same kinds of things.
3. My main discomfort with longtermism has long been that there's something that feels kind of imperialist, or at least foolish, about trying to determine outcomes for a far future that we know almost nothing about. Much of IIDM work involves trying to get explicit about one's uncertainty, but the forecasting literature suggests that we don't have a very good language or tools for precisely estimating very improbable events. To be clear, I have no issue with longtermist work that attacks "known unknowns" -- risks from AI, nuclear war, etc. are all pretty concrete even in the time horizon of our own lives. But if someone's case for the importance of something relies on imagining what life will be like more than a few generations from now, I'm generally going to be pretty skeptical that it's more valuable than bednets.
4. My own career direction has shifted pretty radically over the past five years, and EA-style thinking has had a lot to do with that. Even though I stand by my position in point #1 that cause neutrality shouldn't be a prerequisite for engaging in EA, I have personally found that embracing cause neutrality was very empowering for me and I now wish I had done it sooner. It's something I hope to write more about in the future.

One of the realities of consulting is that, unless you get very lucky,  you generally do have to be at least somewhat opportunistic in taking projects early on. I'm now in the fourth year of running my business and I'm able to be a lot pickier than I was when I first started, but if I limited my work to clients that were only focused on typical EA cause areas, I'd run out of clients pretty quickly. So I've cast my net quite a bit more broadly, which not only expands the opportunity set but also hedges against me getting typecast and positions me to be competitive/relevant in a wider range of professional networks, which I think is valuable for all sorts of reasons.

Another thing to keep in mind is that I've found that having clients that look great on paper doesn't always mean that you are able to achieve a lot of impact with them. Some of my most successful projects have been with clients that did smaller-scale work or were less sophisticated in their approach, because they knew they needed guidance from an outside expert and were willing to cede a lot of authority and creative input to me as part of the process. When you're really trying to innovate and move the field forward, it helps a lot to have clients like these because they aren't anchored on the usual ways of doing things, which makes them more open to trying out ideas. A lot of the sales process for consulting comes down to reassurance that someone else has done this thing and it worked out great for them, so getting those first few case studies locked down can be really important.

Besides theory of change, which tessa mentioned,  I've found myself increasingly focusing on the "front end" of decision-making rather than very detailed tools to choose from among defined alternatives, because in my experience  leaders and teams generally need help putting more structure around their decision-making process before they can engage productively with such methods.

One innovation I've been working on is a tool called the decision inventory, which is a way for clients to get a sense of the landscape of decisions facing them and prioritize among those decisions. It's a much more intuitive exercise and can be done much more quickly than a formal decision analysis or cost-benefit model, so it lends itself well to introducing the concepts and building buy-in among a team to do this kind of work. It can be especially helpful for teams because different team members have a different view of the decision landscape, and will have different ideas about what decisions are important for which reasons, so activating that collective intelligence can be educational for leaders.

1. You should hope that the transition will be painless, but prepare for it to be really, really hard just in case. I definitely recommend starting out with at least 6-9 months of runway for basic living expenses so that you can manage stress about being able to support yourself. It also helps if you can have one or two client engagements lined up before you actually make the jump. In retrospect, I did this transition in Hard Mode by switching cause area focus at the same time as I went from being an employee to an entrepreneur, which necessitated essentially rebuilding my network from scratch. Don't do this. If you do want to make a career switch, you'll have a much easier time if you get another job in your preferred area first and then go independent after that.
2. One thing I've learned since I started is that client work is itself the best business development. There's really no comparison between a pitch and a referral -- the latter is dramatically more effective in making the case. Another tip is that you can create a lot of opportunities by doing the legwork of chasing obscure RFPs for projects you want to do but are not really qualified for, and then approaching other consultants (who are qualified for them) to ask if they want to partner with you on a bid. That way you get to know that firm and you gain relevant experience if your team wins the contract.
3. This really threw me for a loop my first few years. The money is one thing, but being under-utilized for a while can also be really bad for your sense of self-worth -- and scrambling to meet a million deadlines obviously has its downsides as well. I've generally found the valleys to be more challenging to manage than the peaks, as very few of my projects are so time-sensitive that pushing off a deadline here or there is going to cause a catastrophe. I've found it helpful to maintain an active learning and writing practice as part of my portfolio of activities that can expand or contract to meet the moment. These are things I want to do anyway, and so if I find I have extra time to do them it's almost a blessing rather than something to be bummed about.

As part of the working group's activities this year, we're currently in the process of developing a prioritization framework for selecting institutions to engage with. In the course of setting up that framework, we realized that the traditional Importance/Tractability/Neglectedness schematic doesn't really have an explicit consideration for downside risk. So we've added that in the context of what it would look like to engage with an institution. With the caveat that this is still in development, here are some mechanisms we've come up with by which an intervention to improve decision-making could cause more harm than good:

• The involvement of people from our community in a strategy to improve an institution's decision-making reduces the chances of that strategy succeeding, or its positive impact if it does succeed
  • (This seems most likely to be a reputation/optics effect, e.g. for whatever reason we are not credible messengers for the strategy or bring controversy to the effort where it didn't exist before. It will be most relevant where there is already capacity in place among other stakeholders or players in the system to make a change, whereby there is something to lose by us getting involved.)
• The strategy selected leads to worse outcomes than the status quo due to poor implementation or an incomplete understanding of its full implications for the organization
  • (One way I've seen this go wrong is with reforms intended to increase the amount of information available to decision-makers at the expense of some ongoing investment of time. Often, there is insufficient attention put toward ensuring use of the additional information, with the result that the benefits of the reform aren't realized but the cost in time is still there.)
• A failed attempt to execute on a particular strategy at the next available opportunity crowds out a what would otherwise be a more successful strategy in the near future
  • (This one could go either way; sometimes it takes several attempts to get something done and previous pushes help to lay the groundwork for future efforts rather than crowding them out. However, there are definitely cases where a particularly bad execution of a strategy can poison critical relationships or feed into a damaging counter-narrative that then makes future efforts more difficult.)
• The strategy succeeds in improving decision quality at that particular institution, but it doesn't actually improve world outcomes because of insufficient altruistic intent on the part of the institution
  • (We do define this sort of value alignment as a component of decision quality, but since it's only one element it would theoretically be possible to engage in a way that solely focuses on the technical aspects of decision-making, only to see the improved capability directed toward actions that cause global net harm even if they are good for some of the institution's stakeholders. I think that there's a lot our community can do in practice to mitigate this risk, but in some contexts it will loom large.)

I think all of these risks are very real but also ultimately manageable. The most important way to mitigate them is to approach engagement opportunities carefully and, where possible, in collaboration with people who have a strong understanding of the institutions and/or individual decision-makers within them.