Some readings & notes on how to do high-quality, efficient research

By MichaelA🔸 @ 2021-11-17T17:01 (+76)

Here is the doc version of this, which has some additional comments and which I'll continue adding to occasionally without bothering to edit this post. 

You might also be interested in my similar collections on how to do high-impact research, write/communicate well, manage/mentor researchers, and/or get useful input from busy people. 

I made all of these things in a personal capacity, and they don't necessarily represent anyone else's views.

How to use this doc

• I suggest reading everything in bold
• Other than that, you can skim or skip around as much as you want
• Feel very free to add comments/suggestions, including regarding which resources/tips seem particularly useful to you, which seem not useful, and what other things it might be worth adding
• I put the purpose and epistemic status of this doc at the bottom

What this doc focuses on

• How to increase the accuracy, informativeness, insightfulness, etc. of your research outputs
  • Not taking into account how important the original scope/questions/goals were, or whether anyone actually consumes and understands your outputs
• How to get research done in fewer hours of work and less calendar time
• To some extent, how to (rapidly) “get up to speed” on a topic/field

Some things this doc doesn’t focus on are listed in the following footnote.[1]

Readings - misc 

• Scholarship & Learning - tag on LessWrong
  • I haven’t read most of the posts with that tag, though I vaguely yearn to do so
  • Some posts I especially recommend:
    • The Neglected Virtue of Scholarship 
    • Scholarship: How to Do It Efficiently
    • Costs and Benefits of Scholarship
    • The Best Textbooks on Every Subject
      • I didn’t personally find this very useful since most of the recommended textbooks were on subjects that aren’t very relevant to my work and weren’t available as audiobooks (I find it easier to make time to listen to books than to read them).
      • But this was somewhat useful even for me, and would likely be very useful for people interested in more of the relevant fields and people who don’t have a preference for audiobooks over regular books.
    • Literature Review For Academic Outsiders: What, How, and Why
• Research methods - tag on EA Forum
  • Some posts I liked:
    • How to succeed as an early-stage researcher: the “lean startup” approach
    • How to generate research proposals - EA Forum
• Research advice from Carl Shulman (doc created by Claire Zabel in 2016, according to this page)
• Effective Thesis's collection of Resources
• Advice for New Researchers - A collaborative EA doc
• Resources to learn how to do research
• https://www.charityentrepreneurship.com/research.html (and the pages linked to under “OUR RESEARCH PROCESS”)
• Research Skills Toolbox - CERI
• Collection of Resources on Doing Impactful Research Well - SERI
• The doc version of Readings and notes on how to do high-impact research, which contains things that were somewhat out of scope there and are arguably more in-scope here
• Semi-relevant: Notes on EA-related research, writing, testing fit, learning, and the Forum - EA Forum
• Maybe Independent research - EA Forum
• Maybe some stuff in Readings and notes on how to do high-impact research
• Maybe Action: Write a literature review or meta-analysis

Notes - misc

• I suggest implementing Luke M’s advice from The Neglected Virtue of Scholarship and Scholarship: How to Do It Efficiently
  • But there’s one (fortunate!) way Luke’s advice hasn’t aged well:
    • The EA community now has far more money (I think tens of billions more) than it did around 2011, when Luke wrote those articles.
    • We’re now much more constrained by people with the necessary skills and motivations to do various important types of work than by funding.
    • So in most cases, if someone is doing valuable EA-aligned work or is on track to do so, I would be quite sad if their choices about what to read were substantially constrained by how much some things cost, or if they were spending lots of time finding cheaper ways to get the readings they want. 
    • If that’d be the case for you, consider applying for a grant or contacting me to discuss this.
• In addition to Luke’s advice, I’d add that having calls, meetings, or email exchanges with people who know more about a topic that you can also be a great move and that people probably tend to do too little of this and to leave it too late
  • I think that the case for this is basically the same as the case Luke gives for using things like Wikipedia, textbooks, and overview articles
  • Your notes from these interactions could also by themselves be useful research outputs (before you’ve finished your project, or in addition to your main outputs, or if you end up abandoning your project early)
    • See also Action: Conduct expert interviews
    • See also Notes on "writing someone else's ideas/thoughts up" (if someone who isn’t at Rethink Priorities is interested in reading this doc, let me know and I’ll make a slightly redacted version)
  • (I’ve started a doc on Readings and notes on how to get useful input from busy people, but it’s pretty sparse so far.)
• Consider making Anki cards or using some other system for spaced, interleaved, retrieval practice
  • See also Augmenting Long-term Memory
  • See also Suggestion: Make Anki cards, share them as posts, and share key updates
• Consider learning a bit of cognitive science or educational research relevant to how to learn effectively and efficiently
  • E.g., reading short articles on spacing, interleaving, and retrieval practice
  • See also “10. Learn how to learn” in Evidence-based advice on how to be successful in any job - 80,000 Hours
• Consider making a project plan
  • See also Research project planning templates/resources [shared]
  • (For people at RP, this is compulsory for projects that will take >3FTE weeks, and this template is used.)
• Consider thinking explicitly about what your research methodology will be, whether there are standard methodologies and best practices you should use, etc.
  • I don’t think this is essential
    • I’ve rarely done it myself, and I have reason to believe my research (or research-ish writing) has provided value.
    • But I do expect it will often be very useful to have thought explicitly about methodology, to consciously use best practices, etc.
      • And it’s plausible I should do so more often myself
• Consider looking back at your project plan at some points during the conduct of the project
  • I’d suggest doing so at least once, approximately halfway through the project
  • Aims:
    • Notice ways in which you’ve drifted from your original intentions and timeline, and think about whether you endorse those drifts or whether you should change your weekly goals to better align with the original plan
    • Remind yourself of important things that should guide how you go about your work (e.g., what your precise aims were)
• Consider engaging in some retrospective/reflection process after the completion of the project
  • For people at RP, this is mandatory for projects that took >3FTE weeks.
    • See also Suggested process for retrospectives at RP (but that doc is accessible only to RP staff).
  • I personally use this template.
  • This could involve just thinking, jotting down thoughts with no template/prompts, jotting down thoughts guided by a template/prompts, and/or having discussions with other people (e.g., a meeting with your manager)
• Consider also jotting down retrospective/reflection-style thoughts during the conduct of the research project, as and when lessons learned for next time come to mind
• Consider taking notes in Roam, or some other tool that allows infinite indenting (e.g., workflowy)
• Make sure you get feedback at various stages in the process, including fairly early on
  • Consider implementing the advice from the post Giving and receiving feedback, the comments on that post, and the post Asking for advice
  • Feedback can be especially useful if it’s from someone who has a fair amount of context on you and your project, knows more than you about relevant topics, has good judgement in general, has good research taste in general, or is somewhat “skeptical” or “negative”
  • Consider sharing your project plan, an outline of a post, or similar for feedback
    • As opposed to not getting feedback till you’ve already invested significant effort and the general “shape” of the project is fairly set
    • This way people can give input on “early slices”, rather than giving detail-oriented input on closer-to-finished stuff without having seen and given input on that stuff at a higher-level earlier
  • (I probably have other useful tips regarding getting feedback, so maybe leave a comment if you’re interested to hear more in general or if you have specific questions. I’m just not taking the time to rack my brain right now.)
• You should probably use a task management system and some of the principles from Getting Things Done
  • Personally, I use todos in Roam
  • Trello and Asana also seem like good options

[There are probably various other things I could say that might be useful to some people, and I might think more about that later.]

Purpose & epistemic status of this doc

There are probably literally thousands of existing resources (and collections of resources) covering the topics covered here, and I’ve engaged with a very small fraction of them. So in some ways it feels silly to make my own one. 

But after my first experiences managing or mentoring junior researchers, I realised that their projects had gone less well than they could’ve because I hadn’t thought to provide various links or bits of advice that I already knew of. This made me think that creating this doc would help me be a better research manager/mentor, since I could use the doc as a sort of checklist to see what links/advice I should share, and/or just share the doc as a whole. Hopefully the doc will also be useful for me and for people I don’t manage/mentor. 

But maybe people other than me would be better off reading some other resource or collection of resources. That said, this doc will itself link to all relevant things that I know of, think are potentially useful, and remembered to add.[2]

Footnotes

[1] Things this doc doesn’t focus on include how to:

• Do high-quality, efficient research in a particular field or on a particular topic
  • I.e., this doc is mostly generic, rather than tailored to whatever you happen to want to work on
• Come up with important research questions, pick which ones to pursue, come up with a "theory of change" for your research, or assess your impact
  • For those topics, see Readings and notes on how to do high-impact research
• Manage, mentor, or train other researchers, found or run a research organisation, or similar
  • For those topics, see Collection of collections of resources relevant to (research) management, mentorship, training, etc.  
• Build career capital for further research roles, or how to use research activities to gain career capital for other roles
  • Some thoughts on that can be found here: Notes on EA-related research, writing, testing fit, learning, and the Forum
• Work out who you should communicate your research to, or other ways of increasing the chances your findings are acted on in appropriate and impactful ways
• Write/communicate well
  • For that, see Readings and notes on how to write/communicate well
• Be productive in general

[2] Further epistemic status info:

• I intend to not spend much time on this doc. I don't explain why each reading is relevant, I haven't spent a long time thinking about the ideal way to organise this doc, and I expect this doc omits many great readings and tips (including ones I know of but forgot to add here).
• As of July 2021, I’ve done ~6 FTE months of academic research (producing one paper) and ~1.25 FTE years of longtermist research at EA orgs.
• This doc expresses my personal views only.