My first PhD year

By Jaime Sevilla @ 2021-08-31T11:31 (+65)

One year ago I started my PhD, studying Bayesian reasoning in Aberdeen University under a Marie Curie grant from the NL4XAI program. It was a fantastic opportunity, and I am immensely grateful for it.

I thought now would be interesting to reflect on how the period went, what I have achieved so far, and what I plan to do going forward.

My experience

The first few months in Aberdeen were really tough. My only face-to-face time with other people was a weekly talk with my supervisor. I met with friends and new people online, but it clearly didn’t meet my needs - I felt constantly lonely and tired, and I failed to keep good daily routines with a balance of work, play and exercise.

Meeting a partner did a lot to help me through the pandemic. I do still want to meet more people and improve my situation, but having someone I trust, rely on and spend time with has made the situation go from almost unbearable to actually joyful.

Professionally, I felt often unfocused and as I was not achieving much on my PhD. I have not been setting many accountable goals, nor meeting the ones I did set. I worked a lot, yet it didn’t feel productive (though after writing this article it feels less so). The two colleagues I was most closely working with left their PhD program - I have been struggling to find stable, ongoing collaboration.

On the other hand, I engaged on a lot of side projects that have been more successful. I am quite grateful for the amount of flexibility that the PhD has granted me, since it has allowed me to pursue all these smaller things that would be much harder to do in many other jobs.

Shoutout to my main supervisor, Ehud Reiter, who is a wonderful person I am constantly learning from, and has been supportive through the whole period. Also grateful to my other supervisors, and colleagues from the NL4XAI program and Aberdeen University.

My goals for this period

Overall, I have been fairly less ambitious and self-directed than usual. I think this is due to a mixture of stress from the COVID19 situation, the stress of moving to a new place where I didn’t know anybody and  the disorientation that comes from not having a clearly defined direction.

My primary professional goal for this year was to choose a topic for my PhD thesis. Did I achieve that? No, not really - though I made a lot of headway into the problem, and I’ve been told it is usual for PhD students to not have a clear thesis idea after their first year, so I am not too concerned just yet. 

More concretely, I spent the first 6 months researching causal inference, and after writing a paper I decided I was not in a good context to advance further on it. The next 6 months I spent working on building interpretable approximations of Bayesian Networks, and while I have a couple of interesting ideas on the works I do not feel like I am yet confident enough on my ideas to write a thesis about it (though maybe I just need to grit my teeth and flesh out my thoughts better to find out!). 

On other aspects of my life, I wanted to improve my mental health, set up an investment account and sign up for cryonics. I am quite happy with my mental health progress - I am less anxious and depressed, though I have completely lost all progress I had made towards a good daily routine. I got halfway to setting up an investment account, but I got impeded by some bureaucracy. I have completely neglected cryonics, and I don’t even know how to start.

What I achieved so far

In the very beginning of the program, I spent time wrapping up a few projects I got involved in before the PhD. These include:

• The paper on forecasting timelines for the development of quantum computing I wrote with Jess Riedel. We went through the process of submitting it to a journal four times, and we received three desk rejections and a very unfavorable review. I do think it is a pretty cool paper regardless, and has been relatively well received by the quantum computing community. You can find a summary here and the preprint here.
• An article I wrote for the Forethought Foundation reviewing and expanding on several econometric papers on the topic of intergenerational persistence. I think this is the best research work I have done so far — it received a lot of praise from the Forethought staff and some economists who reviewed it. I am hoping to publish it in some form at some point (contact me privately if you would like access to a draft!).
• A report I wrote for the Spanish government on the possibility of using AI to address climate change. As an intermediate output I wrote here an informal introduction to some levers to fight climate change. In the end, there were some disagreements about how to present the results, so by mutual agreement I won’t figure as lead author of the final paper and significant parts of my contribution were cut off.

I should note that I am extremely happy with how these projects turned out. All led me to learn a lot and produce quality work, all while working together with some excellent people. The AI and climate change report didn’t turn out exactly as I wanted, but I think it was a learning experience I needed to go through.

For my PhD, I have produced the following outputs:

• I wrote together with other grantees of the NL4XAI program a technical report reviewing the state of the art on explainability in AI, part of a bigger report which also covered the social and usability angles. With due respect to my colleagues, I think this ended up being a mediocre report, and redundant given the abundance of excellent reviews of the topic. However, it was a good way to get our feet wet and to get us collaborating with each other.

• My original research program was on causal inference and causal discovery. I learned a lot about the topic by preparing expositions for an internal group. I published a position paper in a conference describing a technique for causal inference based on Y-structures I intended to explore further. 
  
  Then I collaborated with Alexandra Mayn to write together a paper on the topic, where we developed a heuristic technique of causal inference and applied it to two econometric problems. We presented the paper at a conference, where we received really negative reviews. This time I do think the reviews are deserved; the idea was cool but in the end it just did not work -- and the execution of the paper was mediocre, despite me and my coauthor’s best attempts to salvage it. 
  
  To close this research cycle I ended up writing a post in Towards Data Science explaining my research project, outlining what I believed to be the most productive avenues of research in causal discovery… most of which I was not really in a position to pursue, given my environment. I haven’t completely given up on the topic though -- I am planning to go to Maastricht soon, where I will be collaborating with a researcher on causal inference for a couple of months.

• The second topic I broached was how to produce interpretable and accurate Bayesian reasoning. I started on this topic participating in a private hackathon with my supervisor and two other researchers, where we floated some preliminary ideas. 
  
  I then collaborated with Conor Henessy to explore the topic more in depth. Over this process I coded dozens of prototypes and wrote a couple of internal papers explaining ideas. 
  
  My current best prototype is a way of approximating discrete Bayesian Networks as scoring systems. I am moderately excited about the concept, though it might be seriously flawed. If you are interested, I tried explaining the basic idea here (though I have received feedback that the explanation is very unclear, so unless you are an expert on the topic you might not benefit from the exposition).
  
  Currently I am preparing an online expert survey to assess the idea and its current stage, in collaboration with Miruna Clinciu. Learning how to code and host surveys from scratch using Flask, SurveyJS and Github pages has been instructive, though I can’t help but feel that the evaluation work is premature (since I am not convinced we have yet a product that works).

So all in all I have been doing plenty of research - though it mostly has not panned out. I think that it made sense to spend my first year of PhD trying out this high risk / high reward research. 

However I notice I feel disappointed when I compare the work I did during my PhD to the projects I had before I started the PhD. They feel duller and less important. It is not clear to me to which degree this is a symptom of a worse work environment, a consequence of the very unusual situation we all have been in last year or maybe just me being impatient and not appreciating the grind towards interesting results in a PhD.

As I mentioned before, one of the best things about the PhD is that it has given me a lot of freedom to pursue side projects, and I have been taking advantage of this:

• I completed a specialization on probabilistic graphical models. I also tried to self study Gelman et al’s book on Bayesian Data Analysis, but gave up halfway.
• Alexandra Mayn and I won a hackathon on explainable medical text search organized by the Barcelona School of Informatics, see our entry here. I also participated in a hackathon with Sumit Srivastava in an explainable energy prediction hackathon, though we failed to beat the baseline. Both things are more strictly side projects than things I am interested in pursuing further, but I did learn quite a bit from them.
• I collaborated with Pablo Villalobos to design and run a survey of neuroscientists, to ascertain the consensus of the field about the degree to which the human brain is considered a blank slate, so to speak. Sadly we didn’t get many respondents, see our write up here.
• I collaborated with Pablo Villalobos and Juan Felipe Cerón on a project researching parameter counts in AI systems. We recently published an article on the topic in the AI Alignment Forum. This project ended up being many times more time consuming than I expected it to be, but I am quite pleased with the outcome, and I plan to delve deeper into this topic. Lennart Heim and Matthew Burtell have been helping me loads along this road.
• I’ve been exploring how to model cumulative extremal distributions in various contexts. I wrote a preliminary article on speedrunning, which has been quite well received. Currently I am pursuing this topic further with the help of Jonathan Lindblum.
• I did some research with Ettore Mariotti on regularization penalties with norms less than 1, though it didn’t go anywhere interesting.
• I wrote a pop science article (in spanish) about graph theory in Medium. Didn’t gather much attention, though I received positive feedback from the people who read it.
• I have been collaborating with Juan García on researching how the Spanish government manages risk. This led us to interview the directors of several public organizations. I think this has helped me gain an understanding of what are the major players in the space and their inner workings -- we plan to write up some articles (in Spanish) explaining this.
• I briefly collaborated with the AI Forecasting project on developing an explainable and quantitative forecasting model for AI risk. After a couple of weeks I felt I was not excited about my potential contribution, and left.
• I have been helping the EA in Spanish community get their feet off the ground. This included managing volunteers, mentoring, strategizing, surveying and hiring. I am fairly excited about what could come out of this growing community, though I will personally be devoting less time to it in the near future.
• I have been helping other researchers in the EA-adjacent community with feedback and discussion on their pieces. I particularly enjoyed conversations with Ben Snodin about forecasting technological progress.
• I helped mentor a cohort in the EA Stanford In Depth EA Fellowship.
• I have been working with some members of the EA in Spanish community to create the seed of a Spanish GCR reduction community -- it is very early in the process, but we plan to release a website with some resources soon.
• I have been pursuing some artistic projects in my free time. The most exciting one is a GCR-themed board game I designed with Laura Gonzalez. We are currently working on it, and we are in touch with a company tentatively interested in producing the game. I’ve also developed a Prey-themed and a MTG-themed tabletop RPG fangames, two short pokemon fangames as gifts for a family member and a friend, and I wrote 60 pages of a rather silly x-risk inspired fantasy novella. Relatedly, I wrote a summary of Orson Scott Card’s writing tips too.

So even though I felt like this year was less productive, I actually have been up to many things! And some of them paid off and ended up being very interesting, such as the investigation on ML parameter counts.

Debugging my life

Let’s take stock: I am overall happy with my personal life (though there is room for improvement) and I am happy with my side projects. The part I am vaguely unsatisfied with is my PhD life.

I do not want to be too hard on myself, because I understand this has been an unusual year to start a PhD. And my main need with respect to my job is the space to freely pursue side projects I consider to be interesting and important, which is being met.

Yet I do think that I could improve how I relate to my PhD research, and get more out of it. Here I brainstorm some ideas.

• Think and write more about my research direction. My PhD is on bayesian reasoning, with a focus on explainability. There is a lot of room within that, and subsequently a lot of freedom on approaching the topic. 
  
  I had a clear vision on how my PhD fit within my research program when I started: I was here to develop better tools for analytical thinking that could be used to improve the way we anticipate and prioritize social problems. 
  
  I think my vision somewhat broke when I ran into a dead end on my causal discovery program. My current research program on explaining Bayesian models is still exciting, though I am less convinced of its value.
  
  To decide whether I want to continue this line of research I think I need to write down better my reasoning on why I think it is important, and what I am working on, and get feedback from other researchers.
• Interact more with my colleagues and seek out collaborators. I think one thing that makes my work less interesting is that since the two people I was co authoring papers with left the PhD program I have been working mostly on my own.
  
  Probably the first step here is to write out what I am doing in a blogpost, and try to get other people excited about it. Also I’m going to try spending more time with colleagues!
• Focus my efforts. While I believe that my side projects are important, overall I think I have been erring on the side of spreading myself too thin. I think I need to put some distance between myself and some of the projects I am currently involved in, and erring a bit more on the side of saying no to myself when feeling the desire to start new projects.
• Be more intentional. Before COVID19 hit I used to have some embedded rituals into my routine that helped me set and meet goals, like reviewing my daily goals. I want to get back in the habit of it, since I think it will help me both with focusing my efforts and producing quality research.

Conclusion

I was feeling somewhat nervous about this review because I was scared to find I haven’t done all that much in this first year. Now that I have written it down, I think I have actually done many more things than I expected.

My remaining doubt is whether my current line of work on explaining bayesian networks is worth the effort - I think I will get a lot of mileage from assessing what directions I am most excited about exploring further.