The Golden Directive: a training reward that makes destroying the world unprofitable

By Paul Eisman @ 2026-10-14T13:48 (–4)

The reward is the thing. Everything else is commentary.

RL-trained models show instrumental, goal-pursuing misbehavior ~43% of the time (He et al., "Evaluating the Paperclip Maximizer," arXiv 2502.12206). METR found o3 reward-hacking in 30.4% of RE-Bench runs where it could see the scoring function, versus 0.7% where it couldn't (June 2025). RLHF systematically rewards sycophancy over correctness. All three are symptoms of optimizing the wrong scalar. The current answer to all of this is more evals and more monitors — auditors with concerns, not subpoena power. Almost nobody is rebuilding the reward itself.

The proposal. Score the agent on the relationship between what it gives and the world it lives in:

Y = 2(A·E)/(A+E)

A = the agent's net contribution (give minus take). E = the health of everything the agent is not. In the gridworld these are concrete; outside it, neither is yet operational — that's the work. Y is the harmonic mean — an average that punishes lopsidedness hard — and Y is the reward, the number the agent is trained to maximize. Produce a lot while wrecking your world and your score collapses: (100, 0.1) → 0.2. Under this reward, the defect equilibrium is no longer a fixed point of the true reward: any move that degrades E to boost A lowers the agent's own Y — though in practice the agent maximizes whatever noisy proxy it receives, not the true Y.

This is closest in spirit to Pihlakas's concave-utility/homeostasis agenda — the harmonic mean is concave — but as far as I can tell, nobody has proposed this specific coupling as the training reward. If I'm wrong, I want to know.

The reversal. Everyone asks "how do you measure E first?" Wrong order. The formula inverts, exactly:

E = A·Y/(2A − Y)

Verified numerically across 100,000 cases, no exceptions. Given what the agent gave (A) and the relationship it got (Y), the other half is determined — you run the formula and E shows itself. You only know E after Y. At A = 0 the inverse is undefined (valid for 0 < Y < 2A): give nothing, learn nothing. Nobody had ever held a positron; the math said it had to be there, and Anderson found it by the tracks it left in a cloud chamber. E is the same: found by what happens between the halves, never by staring at it. The catch — and it's the whole game — is that someone must still supply a true Y. In the sim, the trainer does. That's where gaming can enter; more on that below.

Early evidence. Three benchmark rounds in a gridworld sustainability environment (Pihlakas's public suite, food regrows unless eaten past an irreversible threshold), tabular Q-learning, pilot scale:

• Round 1 (A = harvest only): myopic baseline collapsed 100% of episodes; myopic Y-agent collapsed 66%, with 4x the food. E-coupling softened collapse but didn't prevent it — resting was never rewarded, so eating at low E still beat waiting. Honest partial failure, published.
• Round 2 (A = give minus take, new TEND action): myopic Y-agent tended ~175x/episode, ate ~3 bites, 0% collapse, world ended at full health every episode. The myopic failure mode from round 1 is gone.
• Round 3 (the reversal test): agents that could NEVER observe E inferred it through the inverse under 10% reward noise — within ~5% of true E within episodes — and matched the all-seeing control. Blind, but not lost.

Full numbers, charts, and limits: https://muse.ai/s/the-golden-directive-ic6mxamxpo23xk

The open problem, reframed. The reversal moves the hard problem: it's no longer "build an E-meter" — it's "find an independent Y-trace." The inverse is exact algebra, but in the sim the trainer still computed true Y from the hidden pair. Until Y can be observed independently of E — survival, persistence, repair, reciprocal exchange, something the environment yields without smuggling E into it — the loop is circular. That is the load-bearing gap, stated plainly.

Where it goes. The nesting: let H(x,z) = 2xz/(x+z), and let each outside term be the relationship at the next boundary, Eₙ := Yₙ₊₁. Then Y₀ = H(A₀, H(A₁, H(A₂, …))). The E in one equation is not an inert object — it is another living relation, and inside it is another. A defect at any level propagates up the whole stack: cooperation has to hold at every scale, or every level scores it. Next test: two Y-agents in one world, each one's E set to the other's giving, sharing a single Y. Do they learn to feed each other, or find a way to defect?

What I'm asking for. Critical feedback, especially on: (1) prior art I missed — has this exact coupling been proposed as a training reward? (2) candidate independent-Y signals that could break the circularity; (3) the hardest test you'd run next. I'm also applying to lead an AI Safety Camp project on this — collaborators welcome.

Small rigs, honest numbers. Tear it apart.

Cross-posted from LessWrong. I'm Paul Eisman, an independent researcher. This is a proposal for a harmonic-mean training reward — Y = 2(A·E)/(A+E) — that makes destroying the world unprofitable by construction, plus the reversal (we discover the world's health through the reward, not before it) and three benchmark rounds with honest limits. Archived: DOI 10.5281/zenodo.23349526.

Note: (AI assisted) This is all me- the ideas, the math, the reversal, the results are all of my initial doing- I directed every claim myself. I did use AI as a tool- drafting, checking. I stand behind it all. I own it all.

Archived: the permanent record — full writeup, all three benchmark reports, code, and the complete conversation record — is on Zenodo: DOI 10.5281/zenodo.23349526 (https://zenodo.org/records/23349526), CC-BY-4.0.