Presentation -- The Unjournal: Bridging the gap between EA and academia

By david_reinstein @ 2024-01-22T19:49 (+14)

This is a linkpost to https://www.youtube.com/watch?v=_b5wiSJX6NQ&list=PLwCHmz77VrK1Bi92uyDHE9XCH-bWc-7B_&index=5

Subtitle: Our Approach to Research Evaluation and Impact

Notes and links

Below, a mixed AI & human summary.

Links, unless mentioned, are time stamps to the Youtube video
Presentation slides: here

YouTube playlist: "Unjournal, impactful research, robust open science" (more on this later, building this, may be collaborative)

See Unjournal.org to learn more about The Unjournal. 

Introduction

“What is the Unjournal”

The Unjournal is not a traditional journal. Instead, it represents a new approach to the evaluation of research, particularly focusing on quantitative work that informs global priorities in economics, policy, and other social sciences. This innovative platform is building an open, sustainable system for evaluation, feedback, ratings, and assessment of research projects in any format.

The Problem with Traditional Academic Peer Review

What is academic peer review, what are the issues with it?

In fields like Economics, the traditional publication process can be lengthy, often taking anywhere from six months to a decade. The outcome of this process is typically just the journal in which the research is published, which doesn't provide a comprehensive view of the research's quality or impact.

It also limits the formats of research content, diverts a lot of effort towards gaming the system, and can discourage continued improvement to long-term robust research projects (“I already ‘got’ the publication so who cares”?). 

I come back to this later in the presentation (here) under ‘why not just use academic publishing?’ (slides link here) 

The Unjournal's Solution

“Unjournal’s main ingredients”

The Unjournal addresses these limitations by:

• Research Submission/Identification and Selection: Focusing on impactful research and selecting works that align with global priorities.
• Paid Evaluators: Engaging experts to provide thorough evaluations 
  • … and useful, quantifiable metrics.
• Public Evaluation: Making the evaluation process transparent and accessible to all.
• Linking, Not Publishing: Connecting evaluations to research without the exclusivity of traditional publishing, allowing links to research in any format including web sites and dynamic documents.
• Financial Prizes and Transparency: Offering incentives for high-quality research and maintaining openness in operations.

Theory of Change: leveraging problem synergy

Our theory of change

The Unjournal's approach leverages the synergy between the needs of effective altruism (EA) and the resources of academia. It aims to direct public evaluation of research relevant to global priorities, addressing the limitations of the outdated journal system and shifting attention towards impactful research.

See the flowchart here.

To convince you that The Unjournal has value, I must convince you that…

• Research matters
•  Rigorous prioritization research can positively influence funding, decision-making, and/or policy.
• Rigor and expertise add value. 
• (Peer) review & evaluation can add value to research (and research-use 
• The status quo peer review system is suboptimal 
• Academic publishing has substantial room for improvement and/or Global-priorities-relevant research  would benefit from more scrutiny. 
•  The UJ’s approach can succeed.

What Makes Research Impactful?

“What is it/why does it matter?”

Impactful research is defined as work that produces true, useful information, enabling better decision-making and leading to improved outcomes. This can influence resource allocation, policy nature, and decision-makers' thinking.

How do we prioritize research for evaluation, how do we think about this? See unjournal.org: Broad discussion here, flowcharts referenced can be seen here.

Fields and Approaches Covered by The Unjournal

The Unjournal currently focuses on fields related to human behavior and its consequences, including economics, quantitative social science, business/policy, forecasting, and cost/benefit analysis. It prioritizes empirical measurement, theory/modeling, and methodology with direct policy applications.

Details on our scope on our web page here 

The Value of Evaluation

(Youtube timed-link here … ‘why is evaluation important’?)

The Unjournal emphasizes the importance of evaluation in providing credibility, domain expertise, and usefulness to research users. It also plays a crucial role in prioritizing research for funding and attention.

EA research evaluation …  my impression of ‘the state of the art in EA’ and the limitations, opportunities for wins; benefits coming from the Unjournal’s pilot evaluations

Overcoming Academic Inertia, collective-action problems

The Unjournal aims to overcome academic inertia by providing a platform that is risk-tolerant, externally funded, and innovative. It seeks to make its evaluations a standard part of the academic review process, offering a public alternative before traditional journal reviews.

“Towards a new equilibrium”

Overcoming the CAP

Progress and Roadmap

The Unjournal’s progress / Slide link

The Unjournal has made significant progress in establishing its platform, building a team, and developing systems for research prioritization and evaluation. The roadmap ahead includes raising awareness, establishing credibility, and scaling the scope of operations. This involves building tools and systems to grow without compromising quality.

Our Roadmap ahead (see earlier writeup here)

Challenges and Pivotal Choices

The Unjournal faces several challenges, including gaining commitments from prominent academics and open science organizations, encouraging submissions of dynamic documents, and ensuring that evaluations are recognized and valued in academia. Key questions revolve around evaluation criteria, aggregation, and creating useful outputs for research users.

Slide link

How to Get Involved

How can you get involved? (video link)?

Slide link, links links that you can use to link up and engage

The Unjournal encourages engagement from the academic and research community. Opportunities include:

• Submitting research for evaluation.
• Reading, using, and citing evaluations.
• Spreading the word within academic circles and on social media.
• Partnering with The Unjournal for events and collaborations.
• Joining the team as a paid evaluator or field specialist.
• Suggesting work for evaluation.
• Providing feedback and participating in discussions on platforms like the EA Forum.

Q&A

The Q&A (time stamped link) may have touched on

• How does The Unjournal ensure the quality and expertise of its evaluators?
• What strategies does the Unjournal use to encourage researchers to submit their work for evaluation?
• How does The Unjournal plan to integrate with or complement existing academic publishing and evaluation systems?
• What are the long-term goals and vision of the Unjournal, especially regarding its impact on academia and research?
• How does The Unjournal handle conflicts of interest or biases in the evaluation process?
• What measures are in place to ensure the transparency and accountability of the Unjournal's evaluation process?
• (How) does The Unjournal plan to expand its scope to include a wider range of research fields or disciplines?