Unjournal Evals: "Advance Market Commitments: Insights from Theory and Experience"

By david_reinstein @ 2023-03-21T16:59 (+27)

This is a linkpost to https://unjournal.pubpub.org/pub/amcmetrics

Overview in bullet-point form

The original paper: Advanced Market Commitments: Insights from Theory and Experience by Michael Kremer Jonathan D. Levin Christopher M. Snyder

• A. NBER link; B. 1-click link
• Advance market commitments for vaccines/drugs: practicable policy, seems highly relevant to both global health and development priorities, catastrophic risk  pandemic risk 
• Super-prominent authors (e.g., Kremer=Nobel), Snyder promoted related work in NYT op-ed
• Paper was in AER P&P, not fully peer-reviewed
• Authors present the economic theory case for AMCs (model in appendix), differentating technologically close and distant  products
• Make specific empirical claims about the value generated, based on  'spreadsheet empirics'/BOTECS comparing to alternative vaccines not under AMC
  • E.g., " if PCV coverage in GAVI countries converged to the global rate at the slower rate of the rotavirus vaccine ...  67 million fewer children under age 1 would have been immunized, amounting to a loss of over 12 million DALYs." 

My evaluation summary

...  is largely a 'behind the scenes' of this process

• Why we chose this (see earlier post)
• Authors' positive engagement
• appropriate and very sorely needed for continued and expanded use in global health and disease elimination campaigns. 
• Notes on overlooked  replication spreadsheet linked to AER P&P  supplemental material

All three evaluators generally spoke positively about the paper and gave it good evaluations. See metrics HERE.

^[1]

Evaluation 1: David Manheim

• Mostly very positive
• Wanted more discussion of alternative (non-AMC) approaches to 'theoretically distant targets', outside of 'push vs pull' framing
• Manheim on AMCs: 
  • "appropriate and very sorely needed for continued and expanded use in global health and disease elimination campaigns"
  • and to "accelerate vaccines for known pandemic-potential pathogen threats"  
  • of "more limited" usefulness for novel diseases and technologically more-distant targets; for Covid, the bottleneck was  testing/approval and manufacturing
  • "Other approaches... more appropriate for the most critical future biological risks" (from comment on "relevance to global priorities", rated 60/100)

Authors' response: 

• Agree other mechanisms are suitable elsewhere, including for Covid, link their other papers comparing and discussing these in detail, and give some summary of this
• Discussed AMC here because it was tried at a large scale; "We are not aware of a recent R&D prize of a similar magnitude to compare to."

Evaluation 2: Dan Tortorice

• Brief report
• Recommended further empirical work estimating the benefit, with different counterfactuals
• Suggests future theoretical modeling work considering 'permanence' of AMC, co-financing requirements, a multi-period model, and socially-price-constrained manufacturers

Authors' response : 

• Agree on cautious interpretation of "our empirical exercise"
•  On "how to fund an AMC in perpetuity or...  transition off,” 
  • "the AMC for a pneumococcal vaccine included a provision committing firms to cap their prices close to marginal cost. The idea is that the AMC's top-up payments provide sufficient incentive for R&D and capacity investments even with firms’ charging prices close to marginal cost. Firms’ price-cap commitments ensure that vaccines are affordable in the long-run."

Evaluation 3: Joel Tan 

• Was asked to focus on specific robustness-checking work
• Considered three distinct diseases for comparisons
• "KLS's results should be fairly robust ... the coverage rates for alternative comparator vaccines are in fact lower than the mainline RV comparator KLS chose"
• Suggests other issues that may affect the accuracy of the final estimates of DALYs averted, which he suspects imply some overstatement in net:
  • May overstate gain because it relies on an old estimate and  "DALYs lost per capita to pneumococcus were declining in poor countries year-on-year for two decades even before the AMC"
  • On the other hand could understate it because they "do not model the speeding up of the development of existing vaccines"

Authors' response: 

• General agreement
• "Note that the overall calculation only considers the benefits from the faster rollout of the PCV starting from the observed date of availability, taken as given. It understates the value of the AMC to the extent that the AMC accelerated the vaccine’s development."
  
   

Authors' responses

Generally positive, appreciative, and fairly detailed. See interstitials above.

Final note

There seem to be some obvious next steps to take in robustness-checking the estimates. This would be suitable for an advanced student project or independent research project, which could also publicly build career capital.  

1. ^{^}

   As a semi-caveat to this, note that this was far from 'double-blind': all evaluators signed their work, and the paper came from prominent authors and was known to have been accepted by AER P&P, albeit not a peer-reviewed journal.