Babble!

1. Psychological evaluation
2. Job interview
3. Debate competition judge
4. Using emotion recognition (say by image recognition) to find out consumer's preferences
5. Measuring Pavlov's dog saliva
6. Debate in ancient greek
7. Factored cognition
8. forecasting
9. karma on the forum / reddit 
10. democratic voting
11. Stock prices as an evaluation of a companies value
12. Bibliometrics. Impact factor.
13. Using written recommendations to evaluate candidates.
14. Measuring truth-telling using a polygraph
15. Justice system evaluation of how bad crimes are based on previous cases
16. Justice system use of a Jury
17. Lottery - random evaluation
18. Measuring dopamine signals as a proxy to a fly's brain valence (which is an evaluation of its situation)
19. throw stuff into a neural net
20. python $eval$
21. Discrete Valuation Rings
22. Signaling value using jewels.
23. Evaluation based on social class
24. fight to the death
25. torturing people until they confess
26. market price
27. A mathematical exam
28. A high-school history exam
29. An ADHD test
30. Stress testing a phone by putting it in extreme situations
31. checking if a car is safe by using a crash dummy and checking impact force
32. Software testing
33. Open source as a signal of "someone had looked into me and I'm still fine"
34. Colonoscopy
35. number of downloads for an app
36. running polls
37. running stuff by experts
38. asking god what she thinks of it
39. The choice of a pope
40. Public consensus, 50 years down the line
41. RCT
42. broad population study
43. Nobel prize committee
44. Testing purity of chemical ingredients
45. Testing problems in chip manufacturing
46. Reproduce a study/project and see if the results replicate
47. Set quantitative criteria in advance, and check the results after the fact
48. ask people what they'd think that the results would be in advance, and ask people to evaluate the results afterward. Focus the evaluation on the parameters which the people before the test did not consider
49. Adequacy analysis (like in Inadequate Equilibria)
50. a flagging system for moderators
51. New York Times Best Seller list
52. subjective evaluation
53. subjective evaluation when on drugs
54. subjective evaluation by psychopaths (which are also perfect utilitarians!)
55. subjective evaluation by a color choosing octopus
56. Managerial decisions (a 15-minute powerpoint presentation and then an arbitrary decision)
57. Share-holder reports
58. bottleneck/limiting-factor analysis
59. Crucial considerations
60. Theory of Change model
61. Taking a set amount of time to critically analyze the subject, focusing on trying to find as many downsides.
62. Using weights and a two-sided scale to measure goods.
63. Setting a benchmark that one only evaluates against.
64. A referee evaluating a Boxing match
65. Using score for football
66. Buying a car - getting the information from the seller and assessing their truthfullness
67. Looking at a fancy report and judging based on length, images, and businessy words
68.  

1. peer review in science
2. citations count
3. journal status
4. grant making - assessing requests, say by scoring according to a fixed scoring template
5. evaluating a scoring template by comparing similarity of different people's scoring of the same text
6. Code review
7. Fact-Checking
8. Editor going through a text
9. 360 peer feedback - sociometry
10. gut intuition after long relationship/experience
11. Amazon Reviews
12. ELO
13. Chess engine position assessment
14. Theoretical assessment of a chess position - experts explaining what is good or bad about the position
15. Running a tournament starting with this position, evaluating based on success percentage
16. multiple choice exam
17. political lobbying for or against something
18. the grandma test

It was fun! Hope that something here might be helpful :)

Difficulty ratings in outdoor rock-climbing
Common across all types of climbing are the following features of grades:

• A subjective difficulty assessment of the climb, by the first person to climb it, is used for them to "propose" a grade.
• Other people to manage the same climb may suggest a different grade. Often the grade of a climb will not be agreed upon in the community until several ascents have been made.
• Climbing guidebooks publish grades, typically based on the authors' opinion of current consensus, though some online platforms where people can vote on grades exist.
• Grades can change even after a consensus has appeared stable. This might be due to a hold breaking, however it may also be due to a new sequence being discovered.
• Grades tend to approach a single stable point, even though body shape and size (particularly height and armspam) can make a large difference to difficulty.

There are many different grading systems for different types of climb, a good overview is here. Some differences of interest:

• While most systems grade the overall difficulty of the entire climb, British trad climbs have two grades, niether of which purely map to overall difficulty. The first describes a combination of overall difficulty and safety (so an unsafe, but easy climb, may have a higher rating than a safe), the second describes the difficulty only of the hardest move or short sequence (which can be very different from the overall difficulty, as endurance is a factor).
• Aid climbs, which allow climbers to use ropes to aid their movement rather than only for protection, are graded seperately. However other technology is not considered "aid". In particular, climbing grades have steadily increased over time, at least in part due to development of better shoe technology. More recently, the development of rubberised kneepads has lead to several notable downgrades of hard boulders and routes, as the kneepads make much longer rests possible.

I think climbing grading is interesting as the grades emerge out of a complex set of social interactions, and despite most climbers frequently saying things like "grades are subjective", and "grades don't really matter", they in general remain remarkably stable, and important to many climbers.

Correlating subjective metrics with objective outcomes to provide better intuitions about what an additional point on a scale might mean. Resulting intuitions still suffers from "correlation ≠ causation" and all curses of self-reported data (which, in my opinion, makes such measurements close to useless) but is a step forward.

See this tweet and whole tread https://twitter.com/JessieSunPsych/status/1333086463232258049 h/t Guzey

Simple linear models, including improper ones(!!). In Chapter 21 of Thinking Fast and Slow, Kahneman writes about Meehl's book Clinical vs. Statistical Prediction: A Theoretical Analysis and a Review, which finds that simple algorithms made by getting some factors related to the final judgement and weighting them gives you surprisingly good results.

The number of studies reporting comparisons of clinical and statistical predictions has increased to roughly two hundred, but the score in the contest between humans and algorithms has not changed. About 60% of the studies have shown significantly better accuracy for the algorithms. The other comparisons scored a draw in accuracy [...]

If they are weighted optimally to predict the training set, they're called proper linear models, and otherwise they're called improper linear models. Kahneman says about Dawes' The Robust Beauty of Improper Linear Models in Decision Making that

A formula that combines these predictors with equal weights is likely to be just as accurate in predicting new cases as the multiple-regression formula that was ptimal in the original sample. More recent research went further: formulas that assign equal weights to all the predictors are often superior, because they are not affected by accidents of sampling.

That is to say: to evaluate something, you can get very far just by coming up with a set of criteria that positively correlate with the overall result and with each other and then literally just adding them together.

Winner

Last week we announced a prize for the best example of an evaluation. The winner of the evaluations prize is David Manheim, for his detailed suggestions on quantitative measures in psychology.  I selected this answer because, although IAT was already on my list, David provided novel information about multiple tests that saved me a lot of work in evaluating them. David has had involvement with QURI (which funded this work) in the past and may again in the future, so this feels a little awkward, but ultimately it was the best suggestion so it didn’t feel right to take the prize away from him.

Honorable mentions to Orborde on financial stress tests, which was a very relevant suggestion that I was unfortunately already familiar with, and alexrjl on rock climbing route grades, which I would never have thought of in a million years but has less transferability to the kinds of things we want to evaluate.

Post-Mortem

How useful was this prize? I think running the contest was more useful than $50 of my time, however it was not as useful as it could have been because the target moved after we announced the contest. I went from writing about evaluations as a whole to specifically evaluations that worked, and I’m sure if I’d asked for examples of that they would have been provided. So possibly I should have waited to refine my question before asking for examples. On the other hand, the project was refined in part by looking at a wide array of examples (generated here and elsewhere), and it might have taken longer to hone in on a specific facet without the contest.