EA London Hackathon Retrospective

By Jonny Spicer 🔸, Sam Watts @ 2023-02-17T14:11 (+51)

Introduction

On Saturday 11th February, we ran a day-long EA hackathon in London, modeled off a similar event in Berkeley late last year. This post follows a similar format to the retrospective linked above. You can see some of the basic information about the event, as well as notes on some of the projects that people were working on in this Google doc.

We are incredibly grateful to @NicoleJaneway and @Zac Hatfield-Dodds for their advice before the event, and to Edward Saperia for his advice and assistance on the day itself.

TL:DR;

• We ran a pilot hackathon in London, and were surprised by the success of the event.
• Around 50 people turned up, they gave mostly positive feedback, there were several impressive projects which have plausible impact.
• The event helped build the EA London tech community and generated opportunities for community members to work on impactful software projects on an ongoing, long-term basis.
• We're excited to continue running these kinds of events and want to produce a blueprint for others to run similar hackathons in other places.

Goals of the event

While we hoped that this event would produce artifacts that had legible impact, it was mainly a community-building exercise. Our primary goal was to validate the concept - could we run a successful hackathon?^[1] If so, would running similar events in the future lead to greater tangible impact, even if this one didn't necessarily do so?

What went well

• Approximately 50 people attended, which was more than we'd expected.
• The average skill level was high.^[2]
• The majority of teams had a strong lead or mentor, in most cases an existing maintainer or the person who had the original idea for the project.
• Asking people if they'd like to be put in groups and then doing so generally worked well - I would estimate this was beneficial for 80% of the people who selected this option. See related potential improvement below.
• Dedicating one of the monthly EA London tech meetups to brainstorming ideas for hackathon projects both yielded good ideas and got people engaged.
• Having a show & tell section encouraged attendees to optimize for having something concrete, and gave groups the chance to learn about what other groups had worked on.
• The venue was excellent.
• The average overall rating for the event on our feedback form was 4.36/5 .^[3]

What we could do better next time

• We had a limited budget, and more people showed up then we could provide food for on said budget, meaning we didn't provide lunch after we'd originally said we would. We weren't transparent about this, which was a mistake.
• We didn't do enough to accommodate those who were less experienced coders. In future, we'll use a different question on the sign-up form, along the lines of "how much guidance would you need in order to complete an issue in a project that required coding?". ^[4] We can then organise our groups/projects/activities accordingly, including having more ways to contribute to projects through means other than coding.^[5]
• We underestimated the ratio of people with jobs in the "data" family relative to the "software" family, and so our suggested projects were almost entirely software-focused. 
• We could've had a better shared digital space. This ended up being a bit of an afterthought for us, and we ended up asking people to join a WhatsApp group when they signed in, but it wasn't used much during the day. A different platform^[6] could facilitate more collaboration/visibility between groups, allow people to ask for help more easily, and generally give more of a community feel to the event.

Outputs and resources

• There were several pull requests submitted to existing open-source projects, including VeganBootcamp and Stampy.
• Multiple proof-of-concepts for new projects, including:
  • An LLM-powered summary generator for opinions on the Alignment Forum.
  • An implementation of a reverse-engineering algorithm to visualize the shape of a neural network.
  • GPT-Automator, a voice-controlled Mac assistant capable of completing various complex tasks based off speech input. See the blog posts by Luke and Chidi.
  • A BOTEC calculator for shallow investigations using the ITN framework.
  • Experiments in constraining language models.
• Resources we used in the event (please DM Jonny if you'd like edit access to any of these):
  • Expression of interest form
  • Feedback form
  • Event information doc
  • Collaborative notes on the demos
  • Long, rough list of EA-ish open-source projects
• If you're not already signed up for the EA Software Engineering Newsletter and would like to, you can do so here.

What next

We feel strongly motivated by the outcomes from this event, and think there are a number of ways to build on the momentum from this event, including;

• Similar events run every 2-3 months
• Regular informal "hack nights"
• Themed hackathons, eg one focused on forecasting tools, or an alignment jam
• Helping other organisers run similar events in other places
• Fostering a community of EA software engineers collaborating on EA-related OSS projects

We're keen to define a Theory of Change in order to keep these events value-aligned and impactful. After that, we expect to do more experimentation with event format, frequency etc, as well as ways to encourage community members to work on projects on an ongoing basis, rather than for a single day. 

If you would be interested in organizing a similar event then please reach out as we're happy to share as much knowledge as we can!

1. ^{^}

   Admittedly "success" was poorly defined here. Our version of success was primarily about getting attendees to collaborate on projects in groups, and making progress with said projects.

2. ^{^}

   On our expression of interest form, we asked attendees to subjectively rank their familiarity with programming on a scale of 1-5, with the mean answer being 3.275

3. ^{^}

   n=14

4. ^{^}

   Options would be something like: "a lot", "a little", and "none".

   The actual question we used was "what is your level of familiarity with software engineering on a scale of 1-5?", which in retrospect was too vague. There were only 2 respondents out of 42 that selected '1', which led to the only accommodations being made for folks who put '1' or '2' being placed in groups with people who selected '4' or '5', and all being expected to work on the same project. This left some less experienced attendees feeling like they didn't achieve very much during the day, which was something we were trying to avoid.

5. ^{^}

   Depending on skillsets, it seems best to not be prescriptive about how people contribute to projects, but there are some options for things likely anybody could do, for example QA work or documentation.

6. ^{^}

   Discord seems like a solid alternative. Having multiple channels on the same server would allow people to keep their team's communications in one place, see what other teams are up to, and ask general questions/post memes etc, while keeping each stream of communication focused.