Q1 AI Benchmarking Results: Human Pros Crush Bots

By Benjamin Wilson 🔸, johnbash, Metaculus @ 2025-06-28T17:22 (+8)

This is a linkpost to https://www.metaculus.com/notebooks/38673/q1-ai-benchmarking-results/

By Ben Wilson and John Bash from Metaculus

Main Takeaways

Top Findings:

Other Findings

Data exploration site: You can find interactive graphs for the results of this analysis here:  https://q1-2025-aib-analysis.streamlit.app/ 

Introduction

In Q1 2025, we ran the third tournament in our AI Benchmarking Series, which aims to assess how the best AI bots compare to the best humans on real-world forecasting questions, like those found on Metaculus. Over the quarter, we had 34 bot makers compete for $30,000 on 424 questions with a team of ten Pros serving as a human benchmark on 100 of those 424 questions. Previous quarters had only binary questions, but in Q1, numeric and multiple-choice questions were added to the mix, requiring bots to submit continuous or categorical probability distributions. 

Methodology

In order to test how well bots are doing relative to each other and relative to humans, we set up a forecasting tournament for bots and one for Pros. Additionally, we included some questions from the Quarterly Cup in the bot tournament. The pro and quarterly cup forecasts were hidden from bots so that bots could not just copy the other’s predictions. Each tournament launched a series of questions that resolved sometime during the quarter, and asked participants to assign probabilities to outcomes. We then used our established scoring rule to evaluate participants. During the analysis, we aggregate the forecasts of groups of forecasters to create ‘teams’.

Data Collection:

Scoring: For a deeper understanding of scoring, we recommend reading our scoring FAQ. We primarily use spot peer scores and head-to-head scores in this analysis.

Team selection and aggregation: In the analyses below, we sometimes aggregate a group of forecasters to create a ‘team’. To aggregate forecasts, we took the median for binary predictions, the normalized median for multiple choice predictions, and an average (aka mixture of distributions) for numeric predictions. This matches what the Community Prediction does on Metaculus. Below is how the compositions of the teams were decided:

This analysis generally follows the methodology we laid out at the beginning of the tournament series, with a simplified bot team selection algorithm, which we decided on before knowing the results.

Metac Bots

To compare the capabilities of different LLM models, Metaculus ran 11 bots, all with the same prompt. This prompt is very similar to the one that was used in Q3 and Q4, though updated to support numeric and multiple-choice questions. See the section on code/data for links to the template bot prompts.

We simulated a tournament where these bots competed only against each other (no other participants). This tournament took all the questions from the bot tournament and removed the forecasts of non-metac-bot participants. The bots’ spot peer scores and 95% CI against each other are shown below:

  All the bots used AskNews as a news source, except metac-perplexity and metac-exa. Metac-perplexity used the sonar-pro model, and metac-exa used a custom wrapper around the Exa search API endpoint. In the custom wrapper, gpt-4o decided what to search, what filters to use, got the most relevant chunks from sources, and summarized results. Both these bots used OpenAI’s o1 for forecasting.

From this, AskNews appears to be a better news source for bot forecasting, though confidence intervals overlap somewhat.

These rankings are not very surprising. It seems that models that are ranked better on traditional leaderboards (e.g. o1) are better forecasters as well.

Top Bots

Let's first take a look at the bot-only tournament. This tournament pits all the bots against each other and does not include any humans. You can see the full leaderboard here. Since there are 45 bots, let's first focus on the bots that topped the leaderboard. Below are the top 5 bots when ranked by the sum of their spot peer scores. 

     The best bot was one of Metaculus’s template bots, metac-o1. metac-o1 used AskNews to gather news and OpenAI’s o1 model to forecast. Links to its code are included in the code section below. For reference, here is the prompt that metac-o1 (and the other metac bots) used for binary questions:

"""

You are a professional forecaster interviewing for a job.

Your interview question is:

{title}

Question background:

{background}

This question's outcome will be determined by the specific criteria below. These criteria have not yet been satisfied:

{resolution_criteria}

{fine_print}

Your research assistant says:

{summary_report}

Today is {today}.

Before answering you write:

(a) The time left until the outcome to the question is known.

(b) The status quo outcome if nothing changed.

(c) A brief description of a scenario that results in a No outcome.

(d) A brief description of a scenario that results in a Yes outcome.

You write your rationale remembering that good forecasters put extra weight on the status quo outcome since the world changes slowly most of the time.

The last thing you write is your final answer as: "Probability: ZZ%", 0-100

"""

Our template bots have reached the top 10 in all three tournaments so far (Q3 2024, Q4 2024, and Q1 2025): metac-gpt-4o (previously named mf-bot-1) placed 4th in Q3, and metac-o1-preview (previously mf-bot-4) placed 6th in Q4. Though metac-gpt-4o has also gotten worse over time, getting 17th in Q4, and 44th (second to last) in Q1. 

One conclusion we have drawn from this is that the most important factor for good forecasting is the base model, and additional prompting and infrastructure on top of this provide marginal gains. For reference, most participants did not use o1 in their forecasting due to its cost, so we are unsure whether good scaffolding on top of o1 would provide significant gains to o1. In our next tournament series, we hope to enter a few bots that directly test scaffolding/prompt engineering of different kinds and that stay stable over the tournament length. Please see the section on the “Bot Maker Survey” for an anecdote where scaffolding had a large effect for one of our participants.

We exclude our Metac bots from prize money and so we wanted to give a shout-out, congratulations, and review of our Q1 winner! Our best non-metac-bot competitor and winner of the Q1 tournament was ManticAI, a startup working in the space of AI forecasting! We asked them to share a little bit about their bot, and they mentioned:

We are currently a team of 5. Throughout Q1, we made incremental improvements to our tournament-deployed system, iterating and testing each change. This system does the basics quite well: it has access to some key data sources (news, Wikipedia, economics and finance data, external forecasts) and applies some basic forecasting techniques (e.g. base rates, status quo bias). It is still far from perfect but we are improving it at a good pace. In Q1 we used o3-mini for the prediction head, because this model performed well in our internal testing and was relatively well-priced and fast. The reasoning models seem to give more consistent predictions when resampled with the same information, which is another minor factor in favour. At some point in Q2 we’re planning to replace o3-mini with our own models, from which we hope to get a performance boost.

It looks like ManticAI is building some interesting infrastructure! We look forward to seeing how it plays out at the end of Q2. Also, given o3-mini underperforms o1 in Q2 (as it is doing in more generalized LLM leaderboards), their use of o3-mini to get 3rd place will give some evidence that scaffolding can have an important effect for improving the best models.

For reference, here is the zoomed-out full leaderboard. Like the chart above, this graph is ordered by the sum of spot peer scores, which is used to determine rankings on the site (i.e. so the 1st bar shown took 1st place). However, the y-axis shows the average spot peer score. 

  You’ll notice that there are some bots with an average spot peer score that would have given them a rank in the top 5, except they didn’t forecast on enough questions (and consequently have much larger error bars). We use the sum of spot peer scores since this helps to incentivize forecasting on more questions and reduces our uncertainty about their actual average score.

Bot Team vs Pro Team

To compare the bots to Pros, we used only the 96 non-annulled questions that were identical in the bot and Pro tournament. We then aggregated forecasts of the “bot team” and the “pro team” (see methodology section above).

The bot team that was chosen is:

We then calculated the head-to-head scores of the two teams, along with 95% confidence intervals using a t distribution. The average bot team head-to-head score was -17.7 with a 95% confidence interval of [-28.3, -7.0] over 96 questions.

  The negative score indicates that the bot team had lower accuracy than the Pro team. A two-sided t-test confirmed that pros perform significantly differently from bots (p = 0.0014), while a one-sided t-test demonstrated that pros significantly outperform bots (p = 0.0007).

Below is an unweighted histogram of the head-to-head scores of the bot team. 

  The distribution of scores is generally symmetric with a small left lean and fat tails on both sides. Scores in Q4 were slightly more symmetric, but generally took the same shape.

Binary vs Numeric vs Multiple Choice Questions

We also took a look at how bots did on each question type. When comparing the bot team and the pro team on head-to-head scores, generally, bots did a lot worse at multiple-choice questions (score: -37.5) than at binary (score: -9.8) or numeric (score: -8.7). Generally, the sample sizes in these comparisons are fairly small, so hold this conclusion tentatively.

Here are the average head-to-head scores for the 48 binary questions:

  And here are the average head-to-head scores for the 24 multiple-choice questions (we do not include  confidence intervals because our low sample size prevents us from validating the assumption of normality):

  And here are the average head-to-head scores for the 20 numeric questions (no confidence intervals for the same reasons):

  We also looked at what the results would look like with only binary questions (no continuous, no multiple choice). The leaderboard (ordered by sum of spot peer scores) is a lot closer to the leaderboard in Q4, with some familiar faces up top. Reminder: Pgodzinai, GreeneiBot2, ManticAI, and metac-o1-preview (best equivalent to metac-o1 for Q4) were all in the top 6 in Q4. Notably, metac-o1 loses its first-place lead, and bot-makers get the top spots. Below are the top 5 bots in this modified leaderboard:

 

Team Performance Over Quarters

Now let's compare how the bot team has done relative to the pro team over the last 3 quarters. Below is a graph of head-to-head scores over 3 quarters with 95% confidence intervals.

  The values in the graph above are:

The large confidence intervals do not let us discern any trend. For instance, the true spot peer score of the bots (if we ran on a lot more questions) might be -10 throughout all three quarters. Unfortunately, we just don’t have enough questions to state the direction that bots are trending in (despite having enough sample size to show that pros are better than bots this quarter).

One theory for the decrease in score for this quarter (as compared to previous quarters) is the introduction of multiple-choice questions. As noted above, these were difficult for bots, which weren’t a problem in Q3 and Q4 when all questions were binaries (though some questions were numerics/multiple-choice reframed as binaries).

Another theory is that the pro team has gotten better. They have had a while to practice as a team with each other, and practice improves performance. One other factor is that they have access to better AI tools that can speed up their own research process.

Additionally, as stated previously, we have seen that the metac-gpt-4o bot started in 4th place in Q3, and moved to 44th place in Q1. Assuming that metac-gpt-4o can act as a control, this is evidence that bots have gotten better over the quarters. Some factors to consider:

Therefore, we believe that there is weak evidence that bots have gotten better on binary questions between quarters, and got worse in Q1 due to multiple choice questions. But generally (and statistically speaking), we can’t be sure of anything about how bots have changed over time.

Bots vs Pros (no Teams)

We also compared individual bots to individual pros over the 96 non-annulled questions that they both predicted. We don’t recommend putting too much weight in these comparisons because with enough bots, one will do well purely by chance (i.e. this is a multiple comparison problem). Still, we believe this offers useful context.

Below is a leaderboard ordered by sum of spot peer scores (which is what is used to determine ranks for the AI tournament on the site). The top 17 participants are shown. Average spot peer scores are shown on the y-axis. You’ll notice that if pros and bots competed together, all 10 pros would take 1st through 10th place (with metac-o1 getting 11th place). If we sorted by averages, some bots might be able to take 9th place but have much larger error bars since they answered fewer questions (which is why we rank by sum of spot peer scores rather than average spot peer scores).

  Now let’s remove any non-binary questions. This leaves us with 48 non-annulled questions. Notice how the metac-perplexity bot makes it into the top 10 (6th place), but otherwise, Pros still dominate.

 

Bot Team vs Quarterly Cup Team

The Quarterly Cup is one of the most competitive tournaments on Metaculus. Over the course of the quarter, we copied a number of questions from the Quarterly Cup into the bot tournament in order to compare the performance of the Community Prediction and the bot team. We were able to use 27 questions for the comparison. We aggregated all Quarterly Cup participants’ forecasts and compared them to the aggregate of the bot team. This comparison followed the same methodology as the bot team vs pro team comparison above. This means the bot team was decided based on questions not included in the quarterly cup, and the quarterly cup team consists of all forecasters who participated in the quarterly cup. Consequently, the cup team's prediction ends up being essentially the Community Prediction, just without applying recency weighting as is normally done on the site (i.e. only everyone's last forecast before Community Prediction reveal time is used, and these are all weighted the same).

Here is the list of bots that made the team for this comparison:

Spot scores are calculated at the time the Community Prediction is revealed (so bots could not have access to/copy the Community Prediction while forecasting). Below are the scores:

  The cup team (i.e. community prediction) beat the bots with a spot peer score of 4.8. However, the sample size is small, and we cannot calculate error bars using a t distribution since the distribution of scores fails a Shapiro normality test. This is evidence that the Community Prediction continues to outperform bots. It’s probably safe to assume the error bars are large in this comparison, so this is only a weak indication.

Bots vs Quarterly Cup (No Teams)

During Q1, we ran 2 of our template bots in the quarterly cup. This is unlike the “Quarterly Cup Team vs Bot Team” comparison, since these bots forecasted directly in the quarterly cup as a normal participant. This means they forecast multiple times per question, had a chance to update, were scored directly against all human participants via regular peer score, and generally participated like a normal user. There were 617 participants in the quarterly cup in Q1. Scores below are the sum of peer scores (not spot peer scores) for the competition. There were 42 questions in the quarterly cup.

We found some interesting results:

Most notably, metac-o1 ranked in the top 5% of participants. Granted, there is a decent group of participants who only forecasted on a single question, but this is still fairly impressive. Otherwise, like in the bot tournament, metac-gpt-4o placed near last. 

What might be some reasons for metac-o1’s good performance? It’s hard to say what the biggest factor is (whether it's all skill or not), but here are some things to consider. 

There is probably some level of skill needed to get 25th place like it did, but there is good reason here not to trust it as being as good as a 95th percentile human on a question-by-question basis.

While the cup was ongoing, participants tried to forecast the best-ranking bot between metac-o1 and metac-gpt-4o. You can see the question here. The orange bars are metac-o1’s forecasts. Metac-o1 forecast 5 times before submitting an aggregate forecast of its attempts (which is why there are multiple ‘center dots’ every 2 days). Reviewing the question, we find that metac-o1 forecasted its own performance much better than the community, which brought it from 34th place (the resolution value) to 25th after this question was scored. But generally, this is just an interesting anecdote (and there are other questions where it forecasted much worse).

 

Bot Maker Survey & Evaluating Scaffolding

At the end of Q1, we sent a survey out to bot makers. Prize winners were required to fill in this survey, and it was optional for those who didn’t earn prizes. There were 10 responses from prize winners and 1 response from other bots. Below are some insights from this survey. 

How did scaffolding do?

We have been interested to know if prompt engineering and other scaffolding (e.g. computer use tools, code execution, etc) help bots with forecasting, or if the base model is the most important aspect. One way to measure this is to compare the performance of bots to the metac bot that is most equivalent to them (i.e. used the same LLM model). We asked each bot maker what LLMs they used for their final forecast, and what LLMs they used in a supporting role (like research). When running this comparison, one participant clearly stood out.

Twsummerbot (ranked 8th) used gpt-4o in both their final forecast and in supporting roles. They crushed the equivalent template bot (metac-gpt-4o), which ranked 44th. In the bot-only tournament, Twsummerbot got an average spot peer score of 5.51 [95% CI: 10.61, 0.42] while metac-gpt-4o got -5.53 [95% CI: -1.46, -9.60]. The error bars of twsummerbot’s score do not cross over at all with metac-gpt-4o’s score, indicating twsummerbot had a significantly better spot peer score. 

Most prize winners did not stand out in this way, with some having only marginal gains over their equivalent template bot. Others aggregated multiple models in their final forecast, making a comparison difficult.

So let's dive into twsummerbot’s survey a little deeper, which he has graciously allowed us to share in full. Here are his answers:

 

 

Twsummerbot got 9th in Q4, and 8th in Q3. In the Q1 bot tournament, they answered 263 questions (51 multiple-choice, 54 numeric, 158 binary). Their bot relied on agentic tool calling.

It's difficult to say what was most likely to have allowed twsummerbot to pull ahead in this way, but their approach may be worth emulating.

Correlations with Ranks

We ran a few explorations comparing ranks with different questions on the survey. Please note that these trends mostly reflect on prize winners. Only 1 non-prize winner answered the survey, and trends may not extend to others. Additionally, a sample size of 11 is small.

Time Spent: Moderate Correlation 0.434, less time spent = better rank, noisy data

This question was multiple choice (with an “other” option one person used), so we used the midpoints of time estimates and converted them into hours (i.e. 16-40hrs converted to 28hr). We also converted 1 month to mean 160 work hours. We put this in the “Calculated number of hours” column, then looked at the correlation between “Calculated number of hours” and the “Verified rank” column. This resulted in a correlation value of 0.434. Generally, this would indicate a moderate positive linear relationship, meaning that spending more time on your bot may actually give you a worse rank (remember, higher rank values like 44th are worse). On visual inspection of the scatter plot (excluded for privacy reasons), there does seem to be a linear relationship. Though some of these datapoints could easily be outliers, and we wouldn’t be able to tell with this sample size.

Forms response chart. Question title: What is your best estimate for how many total hours (between all team members) have been put into developing your bot?. Number of responses: 11 responses.   LLM calls per question: Weak Correlation of -0.271, seemingly little effect

Next, we looked at the number of LLM calls that were made per question for each bot. This question was also multiple choice (with an “other” option, one person used), so we took the mid-range of the estimates and then looked at the correlation between this value and the bot’s verified rank. Correlation was slightly negative at -0.271, but weak enough that, on visual inspection of the scatter plot, we would say that the number of LLM calls made has little effect on rank.

Forms response chart. Question title: Your best estimate of the number of LLM calls per question?. Number of responses: 11 responses.   Research vs reasoning optimization: Slight leaning towards research

Bot makers were asked, “When building, have you optimized more for research (external information retrieval) or reasoning (processing information given to the LLM)?”. Of the prize winners, 30% leaned toward research, while only 10% leaned toward reasoning. 40% focused “about the same for both”.

Advice from Bot Makers

Bot makers were asked, “What should other bot makers learn from your experience? “. Here is their advice:

Bot A: Garbage in equals garbage out continues to be very accurate for thinking about how well a prediction bot can perform. The research provided to the bot seems to have a big influence on the final prediction result. Even one research item being off or hallucinated can throw off the bot's final prediction a lot. Using median or mean of numerous runs can blunt this a bit, but this can also blunt "extremizing" scores, which tend to do a little better. It's also not clear that making any additional adjustments to the bot's final prediction via manual calibration improves its performance either. The latest models seem to be a bit better than the older ones, so calibration could make performance worse as LLMs improve. There's likely also just dumb luck involved given the variance of LLMs final output, though you can blunt it with mean/median of multiple runs (the same downsides as above still exist though). My best guess, based on what I've seen so far through 3 tournaments, is that single shot runs with the best/latest models (perhaps taking their mean/median results) will be better than any kind of additional work by the developer (like manual calibration, training the bot on past performance, etc). The only thing that would likely improve results beyond this is ensuring the best and most accurate research is provided to the bot for each question.

Bot B: working on predicting the future for 9 months, and then quitting because you realise that a one-liner to blah SOTA model is better than the thing you've been working on for 9 months, is not a sign of failure: it's a sign that you love problems. So yeah, I'm quitting the comp, I'm working on different problems now, but the verbal currency I gained in the past 9 months is not something I could possibly have earned any other way. So Thanks to all the bot makers, regardless of how much info was shared or not shared! i learned a lot more about the past than the future, which was a really unexpected yet welcome outcome.

Bot C: I'm not sure as I don't think I understand the "base" model out there very well anymore. At the end of Q4, I would have said my bot is thoughtful about 1) creating questions and doing detailed research and 2) structuring the prompt to get the LLM to think carefully about the prediction in a process driven way. But I'm not sure my bot does anything really special anymore. On the negative side, my bot had two many technical glitches in Q1 and didn't end up forecasting on lots of questions. So the learning there is you have to work out those glitches and really make sure the thing runs all the time.

Bot D: I think information retrieval is extremely important; look at some examples of the information your bot is using to make a prediction and check for 1.) accuracy, and 2.) completeness. Reasoning models are always getting better out of the box, meaning information retrieval will be the key magic as opposed to special prompting. With that said, it's also crucial to think about model failure and correcting for poor reasoning/question interpretation. At a minimum, multiple predictions can help solve edge-case incorrect assumptions about a question that result in overconfident predictions.

Bot E: We think haircutting predictions is still underrated. We had some bugs in our numerical system that meant we weren't putting much probability mass outside the bounds when they were open and this cost us a lot. So paying attention to the basics has a good ROI. Multiple rollouts is useful for error correction but seems less important than in the past. We don't have much insight on what others are doing in this regard, but we think it makes sense to use different models for different tasks.

Bot F: Keep a better eye on costs and usage. - Rather than having a single model analyse everything, send the data to be analysed as different "Perspectives", then feed that into your judge model - Don't use the builtin framework. It's a pain, and though I haven't switched, I wish I'd just gone with my own thing from the start. - Perplexity really isn't great. Better to roll your own research

Bot G: The input/output/feedback loop was not conducive to post mortem and corrections. Instructions and checklists is a promising direction. Others can build upon our work with specialization into one topic of forecasting.

Bot H: worked: added the functionality to screenshot the resolution criteria links and send that to an LLM for analysis for further context on the exact metric being measured

Bot I: My bot was really bad with the numeric questions, I would recommend handling the binary and the numeric questions completely differently.

Bot J: The newer reasoning models and search tools can now outperform cheaper models with lots of tools and logic

Other Results

Here are some other questions that bot makers answered. Each question had an “Other” option, which some participants utilized:

Forms response chart. Question title: Which LLM model(s) did you use to make your final prediction/answer?. Number of responses: 11 responses.   Forms response chart. Question title: Which LLM model(s) did you use in supporting roles (i.e. not final predictions)?. Number of responses: 10 responses.   Forms response chart. Question title: How did your bot research questions?. Number of responses: 11 responses.   Forms response chart. Question title: What best describes you?. Number of responses: 11 responses.   Forms response chart. Question title: Did you change how your bot predicted binary questions in Q1 compared to Q4?. Number of responses: 11 responses.   How many people are on your team?:

Did your bot use any of the below forecasting strategies?: Here are the counts of how many prize winners used each forecasting strategy

Links to Code and Data

Below are links to code and data you may be interested in:

Future AI Benchmarking Tournaments

We plan to run more tournaments like this one in the future, past Q2, and will announce timing and details as we have them. For updates, occasionally check the tournament homepage or join our Discord and check in on our “build-a-forecasting-bot” channel. You can email ben [at] metaculus [dot] com with questions. Until details are finalized, you can prepare by setting up your own bot in around 30 minutes with this walkthrough and compete in the Metaculus Cup with your bot! If you need budget to experiment, we cover participants' LLM competition costs through our proxy (email us).