How can good generalist judgment be differentiated from skill at forecasting?

Maybe I've misunderstood but in my humble opinion, and limited experience, forecasting is just a tiny tiny fraction of good judgement, (maybe about 1% depending on how broad you define forecasting). It can be useful, but somewhat overrated by the EA community.

Other aspects of good judgment may include things like:

• Direction setting
• Agenda setting
• Being conscious of when you change direction part way through a judgement
• Understanding the range of factors that are important to a judgement
• Knowing how long to spend and how much effort to invest in a judgement
• Brainstorming
• Creative thinking
• Solutions design
• Research skills
• Information processing
• Group dynamics for consensus building or finding challenge
• Knowing who to trust
• Drawing analogies to other similar situations
• Knowing when analagies are likely to be valid
• Good intuition
• Methords for digging into intuitions
• Ability to test and moderate your intuition
• Scenario planning (distinct from foresight?)
• Horizen scanning (distinct from foresight?)
• Foresight and predictions
• Robust decion making
• A range of models of the world which can inform the judgment
• Good heuristics
• Systems thinking
• Self-awareness
• Ability to adjust for unknown unknowns
• Seeking evidence that contradicts the way you may want to go
• Understanding and counteracting other biases
• Understand statistics
• Accounting for statistical issues like regression to mean or optimisers curse
• Making quantitative comparisons
• Weighing up pros and cons
• Other generic decision making tools that can be applied, of which there are lots
• Specific decion making tools applicable to specific situations
• Knowing which of the above is most relevant to a judgement
• Ability to bring all of the above together
• Speed at bringing all the above together
• Preparing for and understating the consequences of having made the wrong judgement
• Ability to relearn and update judement later with new evidence
• Etc

If you're good at forecasting it's reasonable to expect you'll be above average at reasoning or decision making tasks that require making predictions.

But judgment is potentially different. In "Prediction Machines" Agrawal et al separate judgment and prediction as two distinct parts of decision making where the former involves weighing tradeoffs. That's harder to measure but a potentially distinct way to think about the difference between judgment and forecasting. They have a theoretical paper on this decision making model too.

To answer my own question, here is my best guess for how "good judgment" is different from "skill at forecasting."

Good judgment can roughly be divided within 2 mostly distinct clusters:

• Forming sufficiently good world models given practical constraints.
• Making good decisions on the basis of such (often limited) models.

Forecasting is only directly related to the former, and not the later (though presumably there are some general skills that are applicable to both). In addition, within the "forming good world models" angle, good forecasting is somewhat agnostic to important factors like:

• Group epistemics. There are times where it's less important whether an individual has the right world models but that your group has access to the right plethora of models.
• It may be the case that it's practically impossible for a single individual to hold all of them, so specialization is necessary.
• Asking the right questions. Having the world's lowest Brier score on something useless is in some sense impressive, but it's not very impactful compared to being moderately accurate on more important questions.
• Correct contrarianism. As a special case of the above two points, in both science and startups, it is often (relatively) more important to be right about things that others are wrong about than it is to be right about everything other people are right about.

___

Note that "better world models" vs "good decisions based on existing models" isn't the only possible ontology to break up "good judgment."

- Owen uses understanding of the world vs heuristics.
- In the past, I've used intelligence vs wisdom.

Cambridge Dictionary defines judgement as:

the ability to form valuable opinions and make good decisions

Forecasting isn't (at least not directly) about decision-making (cf. instrumental rationality) but just about knowledge and understanding (epistemic rationality).

A bit tangential, but may still be of interest: a recent paper argued that there are two competing standards of good judgement: rationality and reasonableness.

Normative theories of judgment either focus on rationality (decontextualized preference maximization) or reasonableness (pragmatic balance of preferences and socially conscious norms). ... Normative theories of judgment either focus on rationality (decontextualized preference maximization) or reasonableness (pragmatic balance of preferences and socially conscious norms). ... ay rationality is reductionist and instrumental, whereas reasonableness integrates preferences with particulars and moral concerns.

From the post you refer to:

There are a number of sub-skills, like model-building, having calibrated estimates, and just knowing relevant facts.

Calibrated estimates for future events is the goal of forecasting, and while model-building and knowledge are valuable for this, I think they're valuable in other ways, too. I think another component of good judgement is being able to judge which problems to work on in the first place and how much effort and resources to put into them, falling under instrumental rationality. You need to decide which problems to apply your forecasting skills to, and I don't think this is a forecasting problem.

Also, my understanding is that forecasting is specific to predicting possible future events, and would not include having reasonable views on fundamental research questions, e.g. about consciousness, in physics, in normative ethics, etc..

(I suppose you could try to forecast the answers of experts or even hypothetical experts for fundamental research questions, but experts can be wrong, and this seems like a pretty unusual application and ad hoc way to get at fundamental research questions.)