Maybe Anthropic's Long-Term Benefit Trust is powerless

Thanks for investigating!

From my understanding of boards and governance structures, I think that few are actually very effective, and it's often very difficult to tell this from outside the organization. 

So, I think that the prior should be to expect these governance structures to be quite mediocre, especially in extreme cases, and wait for a significant amount of evidence otherwise. 

I think some people think, "Sure, but it's quite hard to provide a lot of public evidence, so instead we should give these groups the benefit of the doubt." I don't think this makes sense as an epistemic process.

If the prior is bad, then you should expect it to be bad. If it's really difficult to convince a good epistemic process otherwise, don't accept a worse epistemic process in order to make it seem "more fair for the evaluee". 

I think "powerless" is a huge overstatement of the claims you make in this piece (many of which I agree with). Having powers that are legally and politically constrained is not the same thing as the nonexistence of those powers.

I agree though that additional information about the Trust and its relationship to Anthropic would be very valuable.

I basically fail to imagine a scenario where publishing the Trust Agreement is very costly to Anthropic—especially just sharing certain details (like sharing percentages rather than saying "a supermajority")—except that the details are weak and would make Anthropic look bad.

Anthropic might be worried that the details are strong, and would make Anthropic look vulnerable to similar governance chaos to what happened at OpenAI during the board turnover saga. A large public conversation on this could be bad for Anthropic's reputation among its investors, team, or other stakeholders, who have concerns other than longterm safety, or might think that Anthropic's non profit-motivated governance is opaque or bad for whatever other reason. To put this another way: Anthropic is probably reputation-managing, but it might not be their safety reputation that they are trying to manage. It might be their reputation -- to potential investors, say -- as a reliable actor with predictable decision-making that won't be upturned at the whims of the trust.

I would expect, though, that Anthropic's major investors know the details of the governance structure and mechanics.

 

This sounds like a job for someone in mechanism design.

Did you reach out to Anthropic for comments/answers?

the Trust Agreement also authorizes the Trust to be enforced by the company and by groups of the company’s stockholders who have held a sufficient percentage of the company’s equity for a sufficient period of time

...

It's impossible to assess this "failsafe" without knowing the thresholds for these "supermajorities." Also, a small number of investors—currently, perhaps Amazon and Google—may control a large fraction of shares. It may be easy for profit-motivated investors to reach a supermajority.

Just speculating here, perhaps the "sufficient period of time" is meant to deter activist shareholders/corporate raiders? Without that clause, you can imagine an activist who believes that Anthropic is underperforming due to its safety commitments. The activist buys sufficient shares to reach the supermajority threshold, then replaces Anthropic management with profit-seekers. Anthropic stock goes up due to increased profits. The activist sells their stake and pockets the difference.

Having some sort of check on the trustees seems reasonable, but the point about Amazon and Google owning a lot of shares is concerning. It seems better to require consensus from many independent decisionmakers in order to overrule the trustees.

Maybe it would be better to weight shareholders according to the log or the square root of the number of shares they hold. That would give increased weight to minor shareholders like employees and ex-employees. That could strike a better balance between concern for safety and concern for profit. Hopefully outside investors would trust employees and ex-employees to have enough skin in the game to do the right thing.

With my proposed reweighting, you would need some mechanism to prevent Amazon and Google from splitting their stake across a bunch of tiny shell entities. Perhaps shares could lose their voting rights if they're transferred away from their original owner. That also seems like a better way to deter patient activists. But maybe it could cause problems if Anthropic wants to go public? I guess the trustees could change the rules if necessary at that point.