Evidence Action is shutting down No Lean Season

By GiveWell @ 2019-06-06T21:34 (+21)

This is a linkpost to https://blog.givewell.org/2019/06/06/evidence-action-is-shutting-down-no-lean-season

This post discusses a set of issues with Evidence Action’s No Lean Season program. No Lean Season is a former GiveWell top charity and GiveWell Incubation Grant recipient. It is now shutting down. Evidence Action discusses its decision in this blog post.

Here, we share a significant amount of detail about this decision and the factors that contributed. Proactively sharing detailed information about a charity’s shortcomings may be unusual, but it is core to GiveWell’s mission. We are dedicated to transparency about our recommendations—the good and the bad.

Evidence Action has reviewed this post, and we’ve discussed our thinking at length with its senior leadership; however, the views expressed are our own. We have been impressed with Evidence Action’s commitment to transparency and continue to support its other work. These updates have not substantially changed our view of Evidence Action; we expect large programs to experience problems, to a certain extent. We believe Evidence Action responded to these problems responsibly, although we have several open questions.

Summary

Evidence Action is shutting down No Lean Season, a former GiveWell top charity that distributed no-interest subsidies to support seasonal migration in Bangladesh.

As we have discussed previously, a study of the No Lean Season program in 2017 found disappointing results; this led to our removal of No Lean Season, in agreement with Evidence Action, from our list of top charities.

In early 2019, Evidence Action’s senior leadership received allegations that a junior employee of the government agency in Bangladesh responsible for approving the No Lean Season program allegedly forged the government approval, allegedly in collaboration with an employee of the program’s implementing partner. The government agency allegedly later asked the implementing partner for a bribe to grant approval of the program. Senior leadership at Evidence Action then began an investigation that was largely unsuccessful in its attempts to learn more due to lack of full cooperation from the implementing partner. Evidence Action terminated its relationship with the implementing partner as a result. Evidence Action’s senior leadership also found that some Evidence Action program staff who worked directly with the partner did not fully cooperate with its investigation and had violated internal Evidence Action policy.

Evidence Action decided to shut down No Lean Season because the cost of finding and supporting a new implementing partner was too high, given the disappointing 2017 study results.

Separately, Evidence Action also informed us of a tragic accident involving migrants from households that had received No Lean Season subsidies. We do not believe this contributed to the decision to shut down No Lean Season, but we are sharing it in this post as the investigation into this accident recently concluded.

We will provide additional information on the following in this blog post:

• We outline below the factors contributing to Evidence Action’s decision to shut down its No Lean Season program:
• The disappointing 2017 study of the program at scale.
• Evidence Action’s termination of its partnership earlier this year with the organization implementing the program in Bangladesh. After learning of the alleged improprieties (referenced above) in February 2019, senior leadership at Evidence Action began an investigation, conducted by external, independent legal counsel. Given the seriousness of the original allegations, Evidence Action also terminated its contract with the partner. The implementing partner largely refused to cooperate with the investigation, and as a result Evidence Action will not reengage with this partner in the future.
• In the course of its investigation, senior leadership at Evidence Action found evidence of an approximately $400 payment by an Evidence Action program staff member that violated its internal policies, as well as contradictory and potentially misleading statements made by some program staff members to investigators. This finding was not material in the decision to shut down the program, as the implementing partner’s lack of cooperation was already known at that point.
• We summarize the findings of the investigation into the accident involving migrants whose families had received subsidies from the program.
• We do not see any of the above issues as a significant update on Evidence Action as an organization. We expect challenges when working in international development, and think senior leadership at Evidence Action responded responsibly to address these challenges. We do retain open questions about Evidence Action’s selection of implementing partners and its process for hiring and evaluating staff. Finally, we and Evidence Action agree that it should continue to strengthen its financial controls going forward.
• Evidence Action expects No Lean Season to have some funding remaining after the program fully closes out. We expect to ask Evidence Action to redirect the remaining funding it received from GiveWell for No Lean Season to Evidence Action’s Deworm the World Initiative, a GiveWell top charity. We (and Evidence Action) will also take into account donors’ preferences for reallocating this funding; we provide instructions for donors who supported No Lean Season to communicate their preferences to us below.

Shutting down No Lean Season

Mixed evidence of impact for program

We removed our top-charity recommendation of No Lean Season after reviewing the results of a large randomized controlled trial (RCT) of its program during the 2017 “lean season.” Evidence Action agreed with GiveWell’s decision.

The study found that the program did not increase rates of migration in 2017, the first year in which the program was implemented at scale. This implied that, at a minimum, the impact of the program was sensitive to details of implementation, and, potentially, the program was not effective at scale. In either case, the results reduced our and Evidence Action’s expectations about the program’s future cost-effectiveness. As of late 2018, Evidence Action had stopped seeking donations for the program but was continuing to operate it and collect additional data on its impact. Evidence Action ran another large trial of the program in 2018 (for which data collection is ongoing in 2019), and we planned to reassess No Lean Season as a potential top charity in 2019, upon receiving the results from that trial.

Investigation into allegations against Evidence Action’s implementing partner

Termination of partnership with implementing partner

In early 2019, Evidence Action terminated its partnership with its partner in Bangladesh; this partner bore primary responsibility for implementing the program. Evidence Action reports that the termination of its implementing partnership accelerated consideration of shutting down the program, rather than waiting for results from the 2018 trial of the program, and tipped the balance in favor of ending it.

According to multiple Evidence Action program staff (and reported to GiveWell by the law firm that subsequently led an investigation into what happened), the implementing partner told these Evidence Action program staff in February 2019 that it had discovered that the government licenses for it to operate the program had been improperly granted, and that there was a possibility that bribes were paid to government officials to obtain the improper licenses. According to these Evidence Action program staff, the implementing partner said that it had approached the government agency to rectify the issue with the licenses and a high-level official asked for a bribe to issue the licenses. The Evidence Action program staff said that the implementing partner had asked Evidence Action for authorization to pay the bribe. The Evidence Action program staff reported this immediately to senior leadership at Evidence Action, who shared it with us and other major donors to the program soon after.

Evidence Action terminated its contract with the implementing partner as a result, shortly after learning of this event. Senior leadership at Evidence Action hired DLA Piper, a global law firm, to lead an investigation into the issue. DLA Piper told us that it was not able to learn significantly more about the circumstances around the allegedly improperly granted licenses because the implementing partner did not cooperate with the investigation; the implementing partner provided only incomplete financial records and declined to participate in interviews. What we know about these circumstances, therefore, is based on DLA Piper’s interviews with the Evidence Action program staff who were in the February meeting; these staff were consistent in their reports that the implementing partner said that a government official had requested a bribe, but they were inconsistent on some other details.

Payment in violation of internal policies and contradictory and potentially misleading statements by Evidence Action program staff

In the course of its investigation, DLA Piper found that some Evidence Action program staff made contradictory and potentially misleading statements to the investigators, including statements about a payment one of them made in 2018 (of approximately $400) that went against Evidence Action policy.

According to the investigation findings, these staff members had learned that the implementing partner’s application to a government agency for licenses to operate the program in 2018 had been questioned and two of the staff discussed hiring a consultant to help with obtaining the licenses. The staff members requested approval from Evidence Action’s finance team to make a payment to hire a consultant; Evidence Action’s finance team responded that such consultant payments were not allowed by Evidence Action policy. The staff members later submitted a reimbursement request to Evidence Action for a cash payment for the same purpose. It is unclear whether a consultant was hired or how the funding was used. Evidence Action’s finance department sent the reimbursement. Senior leadership at Evidence Action has told us that the reimbursement should not have been sent and that its financial oversight practices were not adequate to detect this payment; it expects to make some changes to its controls and finance staff training as a result of this experience.

We are unsure what to make of these findings. The investigation only established that these staff received reimbursement for a use of funds they had been informed was not permissible under Evidence Action policy, and that they made misleading statements to the investigators. There are missing pieces in the story that we are unable to account for (due to the implementing partner’s lack of cooperation with the investigation and conflicting and potentially misleading statements from some Evidence Action program staff). Senior leadership at Evidence Action has taken what we see to be appropriate corrective action in its staffing; we have chosen not to discuss details about individuals. [1]

We understand that charities may on occasion experience malfeasance by staff, and we believe Evidence Action could not have prevented all possible scenarios where malfeasance might occur. We were disappointed to learn that Evidence Action staff appear to have made a payment in violation of Evidence Action policy and that this payment was approved; we are also disappointed that these staff made misleading statements to DLA Piper. We also believe that senior leadership at Evidence Action responded to the issues described above in a timely, thorough manner by investigating what occurred.

While it is disappointing to learn of improper behavior at one of our top charities, our ultimate focus is whether the program accomplishes good in the world. [2] Of course, if we learned of wrongdoing that was not responsibly handled, and/or reflected a large and serious gap in internal controls, that could lead us to remove a top charity from our list. But broadly speaking, we would not expect to be aware of every instance of fraud (nor do we believe it would be cost-effective for most organizations to put in place controls that would absolutely prevent all malfeasance). [3]

From discussions with senior leadership at Evidence Action, we do not believe the improper behavior by some Evidence Action program staff contributed to the decision to shut down the program, though it was uncovered as part of the investigation into the implementing partner.

Decision to end the program

In light of the 2017 RCT results, the need to find a new implementing partner, and the costs of doing so, Evidence Action decided to terminate the No Lean Season program.

We agree with Evidence Action’s decision. This is a nuanced position: we agree with the decision to shut down the program, but we do not believe that the program is “bad” or “ineffective.” Instead, we believe that continued investment in No Lean Season—taking into account all of the challenges this would involve—is unlikely to be one of the best opportunities we or Evidence Action have to cost-effectively save or improve lives. We discuss how the 2017 study changed our assessment of No Lean Season’s cost-effectiveness here.

The program was completed for the 2018-2019 season, with the exception of some subsidy repayment at the end of the season. Data collection and analysis for the 2018 RCT is ongoing and will be completed; we plan to write about the results once they are available.

Cumilla accident

A few weeks before the conversation that led to the termination of Evidence Action’s partnership with its implementing partner, there was an accident involving migrants from households that had received No Lean Season subsidies. We were saddened to hear of this tragic event. This tragedy did not trigger the shutdown of the program, but the investigation that Evidence Action conducted into the circumstances surrounding the accident recently concluded, so we include it here for completeness.

According to news reports [4] (and reported to us by Evidence Action), in January 2019, a coal-laden truck struck a shed at a brick kiln in Cumilla District in Bangladesh. The shed collapsed, killing 13 individuals sleeping inside. Five of the individuals were from households that had received migration subsidies from No Lean Season. Four of those were between the ages of 15 and 17. It is No Lean Season’s policy to only provide subsidies to individuals over the age of 18 and No Lean Season had a number of protocols in place to enforce this policy [5]; senior leadership at Evidence Action told us that it believes that the teenagers did not receive subsidies directly. The teenagers may have migrated independently of the program or individuals from their households who were over the age of 18 may have accepted the subsidies and given them to the teenagers to use to migrate.

Senior leadership at Evidence Action hired investigators to look into the circumstances around the use of No Lean Season subsidies by underage individuals. The investigators conducted a limited investigation (it was limited, at least in part, because the implementing partner did not provide requested documentation or interviews) and concluded that Evidence Action had “robust safeguards” in place for preventing underage migration, including checking birth dates on personal documents before issuing loans. We don’t see good reason to think that the program systematically increases overall risks of this type of accident.

Has our view of Evidence Action changed?

Taken together, the updates have raised questions about Evidence Action (see below). In general, malfeasance at a charity or its implementing partner could lead us to change our opinion of a charity. However, the details of this case have not led us to significantly reduce our confidence in Evidence Action. The decision to scale up No Lean Season was reasonable: high-quality evidence from when the program was operated at a small scale indicated that it had the potential to be cost-effective. Evidence Action decided to scale up based on that evidence, and ran another high-quality study to test the program’s impact at scale. While there were ethical and managerial lapses by some Evidence Action program staff and its implementing partner, as well as a failure of financial controls to catch an improper payment, we broadly believe that senior leadership at Evidence Action responded quickly, with transparency, and responsibly when the issues were uncovered, both to rectify the lapses and to consider how it might improve and prevent such lapses in the future. Overall, our high-level view of Evidence Action is very similar to what it was before we learned of these developments.

We’ve written before that we see Evidence Action as a group we are highly aligned with and that we are excited to support its growth and development (see, for example, here). We have recommended GiveWell Incubation Grants to support Evidence Action’s operations as well as to support its work to develop potential new GiveWell top charities, and we count two of its programs, the Deworm the World Initiative and Dispensers for Safe Water as a top and standout charity, respectively.

We expect large programs to experience problems, to some extent. We think senior leadership at Evidence Action took quick, thorough action to address the situation by launching an investigation and sharing updates with its major funders, as well as terminating its work with its implementing partner and taking corrective action with program staff who did not comply with Evidence Action’s policies.

We have the following open questions about Evidence Action deriving from these developments:

• Evidence Action’s selection of implementing partners. Should Evidence Action do more to vet its partners? Evidence Action has told us that it plans to make changes to its vetting practices as a result of this experience.
• Evidence Action’s processes for hiring and evaluating staff. Evidence Action staff members violated its financial policies. How should Evidence Action improve its processes for hiring and evaluating staff?
• Evidence Action financial oversight. While we think prevention of all malfeasance would be challenging (and may not be the best use of resources), are there cost-effective ways to reduce the likelihood of funds being misused in the future? What changes to its financial controls should Evidence Action implement?

We plan to continue discussions with Evidence Action to better understand its work in these areas.

What will happen with unused funds?

Over the course of operating No Lean Season, Evidence Action received funding earmarked for this program specifically and spent down a portion of that funding. Evidence Action expects to have funding remaining that is designated for No Lean Season when it has fully closed out its work on the program. A large portion of the remaining funding is from Good Ventures, a large foundation with which we work closely, which donated to No Lean Season as a result of GiveWell’s recommendation.

We expect to ask Evidence Action to redirect the remaining funding it received from GiveWell (donations made through our website or to GiveWell via check, wire transfer, or other means), including funding from Good Ventures, for No Lean Season to Evidence Action’s Deworm the World Initiative, which is a GiveWell top charity. If you made a donation to support No Lean Season and prefer that your donation (less the portion of total revenue that No Lean Season has spent) go to an Evidence Action program other than Deworm the World, please contact us at donations@givewell.org by July 31. We will also be emailing donors whose contact information we have. By default, if we don’t hear anything, funding will be directed to Deworm the World.

Note that by “remaining funding,” we mean the original donation size multiplied by the percentage of total revenue for No Lean Season that will remain when the program is fully closed out. We then expect to recommend that Good Ventures reduce its next annual grant to Deworm the World (assuming Deworm the World remains a GiveWell top charity, which we expect it to) in December by the same amount, so that Deworm the World does not receive more GiveWell-directed funding in 2019 than it would have in the absence of No Lean Season’s remaining funds.

We will post an update on our blog about how remaining GiveWell-directed funding for No Lean Season was reallocated.

Evidence Action’s implementing partner holds some program funding, primarily the grant from Evidence Action to the implementing partner for the subsidies for migrants that was collected for the 2018 implementation of the program. It is unclear whether it will return these funds to Evidence Action; it has not yet agreed to do so.

Notes

1. Evidence Action has told us that there is complexity in discussing in general terms the situations with the staff members involved because of the differences in the structure of their employment across several legal jurisdictions.

2. We generally agree with the “results not receipts” approach advocated by the Center for Global Development in this paper.

3. We did learn about two cases of staff fraud at GiveDirectly, another GiveWell top charity, in the past. You can read more in our review of GiveDirectly’s work here.

4. News reports of the accident are available here and here (among other places).

5. From Evidence Action’s blog: “The investigation found that the safeguards we had in place were robust, though ultimately could not fully eliminate the risk of an adult recipient choosing to pass their cash transport subsidy to a teenager in his place, contrary to program rules and protocols. These protocols were multilayered, and included verbally informing subsidy recipients of the condition that migrants must be at least 18 years of age; requiring subsidy recipients to sign or thumbprint an acknowledgement that both recipients and migrants (where different individuals) must be at least 18 years of age; reviewing national identification cards to verify that the subsidy recipient and any person that the recipient says plans to migrate from the household is at least age 18; and utilizing mobile data collection software that is programmed to prohibit field staff from including individuals reporting to be under the age of 18, in order to prevent accidental enrollment.”