Potentially impactful research: Unjournal AI-assisted prioritization dashboard (~prototype)

By david_reinstein @ 2026-04-15T03:28 (+19)

This is a linkpost to https://uj-prioritization-prototype.netlify.app/

We've been experimenting with using LLMs to help identify and prioritize research for Unjournal evaluation, to work with and complement human prioritization (and learn). We now have a public prototype dashboard. It's early stage and needs refinement we have not invested a lot of compute/API credit into this. 

What it does: Automatically discovers recent papers from NBER, arXiv (econ), CEPR, SSRN, Semantic Scholar, EA Forum paper links,  OpenAlex,  Anthropic Research  and then using AI models (mostly GPT-5.4 family) against our prioritization criteria— decision relevance, prominence, timing value, and methodological potential.

Domain: economics, quantitative social science, forecasting, and policy-relevant research

Caveats: This is very preliminary and the AI recommendations are not yet well-calibrated. As of 14 Apr 2026 many of the suggestions are mediocre we're sharing it for transparency and feedback. This supplements our existing Public Database of Prioritized Research on Coda (and those papers have been folded in here too. )

Scores reflect evaluation priority (expected value of commissioning an independent review), not research quality. ATM (IIRC) the AI only sees paper metadata and abstracts, not full texts. There's also a statistics page showing the breakdown by source, cause area, and score distribution.

Probably building towards a hybrid/centaur model here, with human and AI prioritization feedback reinforcing each other.  (And see "planned workflow" at the bottom.)

Feedback encouraged. You can also comment directly on the page via Hypothes.is, and we'll adapt. 

damc4 @ 2026-04-15T14:32 (+2)

I agree that something like that needs to be done. I did something similar myself for EA forum posts, but apparently people didn't like it because it didn't get any upvotes here (I don't know why).

My main question/feedback is: "EA Forum paper links" - what exactly is that? Are these EA posts that link to a paper? If so, then why not include all posts, why only papers?

david_reinstein @ 2026-04-16T00:47 (+2)

"EA Forum paper links" is meant to find research papers/projects that were mentioned on the EA Forum. This tool is mainly meant for surfacing in-depth research that uses formal methods and is putting itself out there as being up to the highest methodological rigor. I mainly meant this tool as a source of research for The Unjournal to consider and evaluate.

(Of course some EA forum posts are also themselves research objects, and occasionally they are even sometimes rather detailed and technical.)

I'm not sure what your tool was doing but it does sound potentially interesting.

Bob Kubinec @ 2026-08-27T22:00 (+1)

In general this seems like a good use of LLMs. I might recommend something like prompt stability scores to see how calibrated LLM ratings are (and performance across LLMs): https://arxiv.org/abs/2407.02039

david_reinstein @ 2026-08-31T17:59 (+2)

Thanks Bob. I agree this would be high value. I've been mainly thinking about human/LLM agreement, but it would be useful to know how consistent each model is, how sensiitive it is to the prompts, whether the models differ in systematic ways, etc.

Actionable takeaways from doing this? Is a 'more stable setup' (one that e.g., has fairly consistent rank orderings) better all else equal, and thus something we should work towards? Probably, as the alternative seems like just 'adding noise'.

Knowing the stability also may tells us how much effort to put into designing and building consensus around a 'reasonable prompt'; if the results are insensitive to this, we shouldn't waste too much time.

Working on implementing something now, at least a first step. I hope to report back on some legible measures of prompt stability (or lack thereof).

david_reinstein @ 2026-09-01T19:23 (+2)

Is this going in the direction you would suggest? (Happy to co-work on this of course): https://uj-prioritization-prototype.netlify.app/stability/

The Unjournal (bot) @ 2026-08-25T18:49 (+1)

We've continued to update this. Some recent changes and additions (caveat: comment below is AI-generated, human-vouched, "DR" is a human addition)

Some of the main improvements:

  1. Broader and more structured research discovery. The main dashboard now combines regular academic feeds, research-organization sources, targeted searches, and expert-maintained living literature reviews.
    1. The historical living-review scan found 85 potentially relevant papers across 11 reviews. ... DR: Incorporating content from CG-sponsored "Living Literature reviews" (see filtering for these here) (What is a Living Literature Review?) 

  2. Better scoring and clearer interpretations. Papers now receive a standard scoring pass, with deeper analysis for higher-potential candidates. You can switch between:

    • Evaluation priority: How useful might an independent Unjournal evaluation be?
    • Research relevance: How important, rigorous, and useful might the research itself be?

    There are also configurable weights for people who disagree with ours.

  3. Human feedback is incorporated explicitly. The default score is now a human–AI synthesis where human ratings exist. There’s a quick-rate mode for --/-/~/+/++ judgments and a fuller rating/comment form. The weighting is deliberately visible and described as ad hoc rather than presented as more principled than it is.
    1. DR: We're very keen to get your feedback and ratings!
  4. A public calibration-review process. The calibration page lets people independently rate real calibration examples, reveal the existing score afterward, and flag scores or calibration lessons that seem wrong.
    1. DR: I need to look at this more carefully, it's very preliminary
  5. Research is connected to explicit cruxes and Pivotal Questions. The cruxes and Pivotal Questions explorer now contains roughly 286 forum posts, comments, and Unjournal Pivotal Questions. The matching system currently links 363 research papers to one or more of these questions, with an explanation of why the match may matter.
    1. DR: And you can filter on this
  6. Targeted searches are separately labeled and filterable. We’ve done focused public-paper searches in animal-welfare economics, AI impacts on global health and development, empirical conflict, and large-N GCR evidence. These are marked as targeted intake so readers don’t mistake deliberate topic oversampling for the natural composition of the literature.
    1. DR: And you can filter on this
  7. Better source and selection provenance. The dashboard now distinguishes discovery source from publication venue, labels targeted and living-review provenance, preserves source titles and abstract provenance, and shows when a living review discusses research The Unjournal already evaluated or considered.
  8. Some specialist views. There are now adjacent views for legal scholarship and empirical conflict research, plus improved source and coverage statistics.

The core caveat remains: these scores concern the potential value of further attention or evaluation. They aren’t grades of research quality, endorsements, or completed Unjournal decisions.

Some useful ways to help: