The State of Metaculus

By christian @ 2025-02-05T19:17 (+17)

This is a linkpost to https://www.metaculus.com/notebooks/34747/the-state-of-metaculus/

This linkpost summarizes Metaculus updates and initiatives from mid-2024 to early February, 2025. The original post covers technical improvements, new forecasting tournaments, and collaborations with CDC, GiveWell, Bridgewater Associates, Bright Line Watch, and more. While written for regular Metaculus users, I'm sharing it as a resource for those interested in Metaculus's recent work and potential ways to participate.

The Metaculus of early 2025 is a different and better place in myriad ways than the Metaculus of a year ago. It can be easy to miss some of these changes and also some of the opportunities — to sharpen thinking (yours and others'), to win prizes and job opportunities, and to contribute to public sense-making in turbulent times. 

This post covers Metaculus from mid-2024 to today, highlighting new developments and features you might have missed — from tournaments and research initiatives to ongoing experiments.

We were fortunate to work with many inspiring partners in 2024, on more projects than we could list below. If you or your organization want to collaborate with us in 2025, get in touch!

As always, feedback is welcome — or you can share ideas in the platform feature suggestions thread. 

A Record-Breaking Start to 2025

Astral Codex Ten 2025 with Scott Alexander has become our fastest-growing series ever with over 3,000 forecasters so far — that's double last year's turnout. Even more dramatically, the return of the Bridgewater x Metaculus Forecasting Contest has registered over 17,000 (!) competitors globally, more than 10x last year's count. The past week has been one for the record books - each day shattered either our record for new forecasters or total predictions made. The Monday launch of the Bridgewater contest topped them all, hitting new all-time highs for forecasters, commenters, and signups.. (And yes, the platform is experiencing some growing pains as we welcome new users. We hope you make folks feel welcome, even as you maybe direct them to the Community Guidelines or Why Forecasting? pages.)

Even before those record-breaking days, your predictions have been reaching ever broader audiences, with coverage spanning major outlets from Fast Company and Vox to Asterisk and AEI posts. We've also been cited by public thinkers like Yoshua Bengio, Vitalik Buterin, and Matthew Yglesias. We've also made inroads with Spanish-language media like El País and with Brazilian papers covering our new Brazil-focused Tournament. 

(Some of) What Our Community is Accomplishing

Here are some of the projects you've contributed to.

Informing Public Health Policy

For the first time ever, the Centers for Disease Control and Prevention included aggregate probability estimates in their 2024-2025 Respiratory Disease Season Outlook. The Respiratory Outlook 2024/25 tournament continues to gather predictions on respiratory diseases and avian influenza risks, with eight Pro Forecasters providing detailed analysis and regular forecasts to the CDC.

From CDC’s 2024-2025 Respiratory Disease Season Outlook

Bridgewater Wants Raw Forecasting Talent

After a highly successful first run that led to multiple job offers for top performers, our $25,000 forecasting competition with Bridgewater Associates returned, launching yesterday. The contest has already exceeded expectations — over 17,000 participants from more than 100 countries have registered. You can read Bridgewater Deputy CIO's op-ed on the Metaculus tournament and on finding innovative ways to spot raw talent in Fast Company, and you can register for the competition here. 

Benchmarking AI's Capabilities on Real-World Questions

Our AI Forecasting Benchmark launched in Q3 2024 to systematically compare AI forecasting capabilities against top human predictors. Q1 2025 recently launched with a $30,000 prize pool and a more rigorous framework that includes a wider variety of question types.

You can enter and learn how to compete here. We offer templates to get you started, and there’s still plenty of time to build and enter your bot! Stay tuned for our analysis on how bot performance in Q4 compared to the pros. 

International Inroads

The 2024 International Elections Hub, launched in partnership with journalist Kiko Llaneras of EL PAÍS, covering outcomes in a historic year where countries representing half the world's population held elections. We're expanding this coverage in 2025 and also developing a Spanish-first forecasting competition with Llaneras that should launch mid-February. 

We also launched our first Portuguese-language tournament with $1,500 in prizes, developed in collaboration with a leading Rio de Janeiro tech journalist. Just in the last few days we've welcomed thousands of Brazilian forecasters to the platform. 

US Election Coverage

Our election coverage centered on two key projects. The Election Hub provided an interactive map for tracking the presidential race with state-by-state predictions and electoral vote counts updated as forecasts evolved. We also launched the Election Consequences project to look beyond vote counts and explore concrete outcomes under different scenarios, like dramatically different probabilities for Russian territorial gains in Ukraine depending on which candidate won. 

Collaborations and Contests

We partnered with individuals and organizations to create focused prediction series:

• The ACX Prediction Contest: We raised the prize pool to $10,000 for this year's annual collaboration with Scott Alexander, and more than 3000 of you already shared predictions on questions supplied by Scott. The competition window may have closed, but you can still forecast, update, and comment to help clarify the year to come. 

• Five Years After AGI: We launched this series in an attempt to look beyond the question of when AGI arrives to understand what a world with AGI looks like. We used our longstanding date of general AI question as the threshold for when AGI is achieved and launched dozens of questions to cover the range of outcomes that matter, including outcomes for employment, economic growth, extinction risk, wealth distribution, and measures of well-being.
• Forecast With GiveWell: Working to improve real-world charity funding decisions, we (along with some very generous Manifund donors) offered a $5,500 comment prize for insights that could sharpen GiveWell's grant outcome expectations.
• Forecasting Research Institute: We launched ten high value of information (VOI) AI questions from FRI's Conditional Trees report, creating a resource for tracking crucial AI developments.

Research and Expert Workshops

We hosted several expert workshops:

• Astera Synthetic Biology Workshop: We partnered with Astera Institute to identify technical innovations needed for scaling synthetic biology companies.
• Threshold 2030: We co-organized with Convergence Analysis, bringing together 30 experts from major tech companies and institutions to assess AI's economic impact by 2030. We'll be publishing our report soon.
• The Curve: We explored AGI readiness through a thought experiment with engineers, safety researchers, and policymakers, leading to a question series driving our experimental AGI Readiness Index.

Security and Public Sense-making Projects

• Day One Project: With the Federation of American Scientists we developed policy recommendations for the Day One 2025 project. You can read the policy memo here: Collaborative Intelligence: Harnessing Crowd Forecasting for National Security.
• SecureDNA Research: Our forecasters contributed to the Institute For Progress's report on Securing Benchtop DNA Synthesizers.
• Taiwan Tinderbox: We launched the Taiwan Tinderbox series tracking potential conflict indicators.
• Holding Public Figures Accountable: And we provided embedded predictions for the Max Tegmark-founded Verity news platform to hold public figures accountable when they made predictions. 

Platform Improvements and New Features

Going Open Source and Site Rewrite

One of our biggest undertakings in 2024 was completely rewriting the site and making Metaculus open source. The results have been exciting: a third of our code contributors are now external to Metaculus, a dozen organizations have forked our repository, and we're fielding inquiries about self-hosted instances. 

Want to contribute? Check out our repository and newcomer-friendly issues – we'll send you a Metaculus hoodie if you make a significant contribution!

This complete rewrite has dramatically accelerated our ability to ship new features. We enhanced user profiles, gave you more control over notifications, and introduced zoom timelines so you can focus on the period in a question's forecast history that you most care about. We also introduced automatic translation into Spanish, Portuguese, Czech, and Chinese, while upgrading our API. Really, there's a lot. Read all about it here. 

And in case you missed it, you can now withdraw your forecasts when you don't want to be on the hook to update them. Finally, new forecasters benefit from guided tutorials that walk them through their first predictions. Just click your username to find 'tutorials' in the dropdown. 

We've also been rolling out more experimental features: Our latest is indexes, which are designed to tackle big-picture questions that tend to resist clear operationalizations. Say you have a question like “Will we be ready for AGI in 2030?” Defining and measuring “readiness” directly is challenging, and defining it clearly threatens to lose key details we care about. So, we’re exploring creating quantified indexes comprised of a number of more clearly defined lower-level forecast questions, which are each given a weight and direction — that is, whether an increase in the forecast raises or lowers the index, and by how much.

You can find our first Index, on how ready we are for AGI, here. 

Oh, and we introduced a "Changed my mind" button to let you quickly update forecasts when persuaded by others' arguments. 

Build Your Own Forecasting Community

Communities are an exploratory feature, providing hubs for bringing forecasters together around a central theme, organization, or figure. Run by volunteers, they lower the barrier for creating forecast questions, enabling exploration of a wider variety of topics with different approaches. We currently host 40 Communities, with most unlisted in the questions sidebar, as they serve specific niche groups.

Interested in setting up your own Community? Message us, and we'll get you set up.

Featured Community: Bright Line Watch — $2,500 Prize Pool

Bright Line Watch, a nonpartisan democracy watchdog group comprising academics from Dartmouth College, University of Chicago, and other institutions, anallyzes key trends in American democratic health. Since 2017, they've been surveying political scientists and the public about democratic norms, election legitimacy, and institutional trust. They've created a series of questions they're posing to 500 academics, and they want to compare the results to the predictions of the Metaculus community. 

Think you can outpredict the experts? There's a $2,500 prize pool for the top forecasters, who'll share predictions on whether Trump will invoke the insurrection act, whether Kash Patel will be confirmed by the senate, whether the White House will revoke media credentials of reporters, and more. 

Forecasts are likely to appear in media and in Bright Line Watch reports. Get started. 

Listed Communities

While most of our Communities are Unlisted, creating spaces for groups like ivy-league forecasting teams, college forecasting courses, niche Substacks, and policy-focused podcasts, some are publicly available in the sidebar, including:

• Vox's Future Perfect team — a series of questions developed by Future Perfect and featuring a $2,500 Prize Pool
• The Brazil Forecast Community
• AI Warning Signs

Some of the Features We're Exploring for 2025

• "Key factors" for forecasts, which can help map out the main drivers of a question
• Making base rates more explicit and shareable, allowing forecasters to collaboratively review and build on them
• Tools to connect forecasts, drawing inspiration from Bayes nets and fuzzy cognitive maps
• A simplified forecasting view for news consumers who need to digest a topic quickly
• A streamlined Survey Flow interface that will make participating in tournaments faster and more intuitive
• Quantile forecasting to let you enter probabilities directly on numeric questions
• Forecasting teams, to make predicting more social and collaborative

As always, we welcome your feedback and new ideas for features. Share your thoughts in the comments here or on the original Metaculus post.