Superintelligent AI is necessary for an amazing future, but far from sufficient

• What is the definition of value are we are supposed to be using  (my current intuition is average CEV of humans)?
• Was I meant to just answer the above question with my own values (or my CEV)?

The OP defines Strong Utopia as "At least 95% of the future’s potential value is realized.", and then defines the other scenarios via various concrete scenarios that serve as benchmarks for how "good" the universe is.

CEV isn't mentioned at that part of the article, nor is any other account of what "good" and "value" mean, so IMO you should use your own conception of what it means for things to be good, valuable, etc. Which outcomes would actually be better or worse, by your own lights?

My own personal view is that CEV is a good way of hand-waving at "good" and "valuable", and I can say more about that if helpful. The main resource I'd recommend reading is https://arbital.com/p/cev/.

I don't know what you mean by the "average" CEV of humans. Eliezer's proposal on https://arbital.com/p/cev/ is to use all humans as the extrapolation base for CEV.

I predict that if you ran a CEV-ish process extrapolating from my brain, it would give the same ultimate answers as a CEV-ish process extrapolating from all humans' brains. (Among other things, because my brain would probably prefer to run a CEV that takes into account everyone else's brain-state too, and it can just go do that; and because the universe is way too abundant in resources and my selfish desires get saturated almost immediately, leaving the rest of the cosmic endowment for the welfare of other minds.)

• What is the definition of cosmopolitian value and how is it action relevant in all of this?

The article for that is https://arbital.com/p/value_cosmopolitan/. "'Cosmopolitan', lit. "of the city of the cosmos", intuitively implies a very broad, embracing standpoint that is tolerant of other people (entities) and ways that may at first seem strange to us; trying to step out of our small, parochial, local standpoint and adopt a broader one."

E.g., someone who values all people on Earth is more cosmopolitan than someone who just cares about the people in their country. Someone who values digital minds is more cosmopolitan than someone who just cares about biological ones.

A lot of where this gets tricky (and where the post directs its focus) is that cosmopolitanism encourages you to tolerate (or even embrace) values diversity in many respects. But if you're maximally embracing of values diversity, then this seems to reduce to having no preferences or priorities at all -- anything goes. So there's then a question, "What would the ideal cosmopolitan value system say about various forms of values divergence?"

I don't really get the purpose of CEV for this stuff or why it solves any deep problems of defining value. I definitely think we should reflect on our moral values and update on new information as it feels right to us. 

If you think CEV is an obvious and prosaic idea, then you probably understand CEV pretty well. :P It's not meant to be anything fancy or special; it's just meant to articulate in very broad terms the sort of thing we probably want to do in order to solve morality (in all the ways that are required for us to steer the future well).

CEV isn't a full specification of morality, but it's a simple, informal articulation of how humanity can attain such a specification (insofar as we need one in order to know what to do).

CEV is agent dependent, so we need to specficy how we weight the CEV's of all the agents we are taking into consideration.

Eliezer's proposal (which seems fine to me) is that we weight all humans equally. "CEV with all humans weighted equally" could then choose to defer to a different CEV that has some other, more sophisticated weighting; but weighting all humans equally at the outset seems fine to me, and has the advantage of being simple and less-likely-to-cause-controversy.

In any case, my main complaint is that if the answers to the above questions are at least in part a function of what our CEV  is(or what definition of value we are use), then I feel like we are stacking two questions on top of each other and not necessarily leaving room to talk through cruxes of either. 

I think I'm pretty used to this because I treat most moral questions as questions about CEV. So, e.g., if we're debating what the tax rate should be on luxury goods, I'm doing my best to estimate what my CEV would think about fairness, compassion, etc. when I make my decision.

Let's assume we are just taking the average CEV of human's alive today as our definition of value. Some vales might be more difficult to pull off then others, as they may trend further from what aliens want or just be harder to pull of in the context of the amount of shards we have.

average -> aggregate

shards -> shard

I think that we'll have way more cosmic resources than we know what to do with, in terms of maximizing the welfare of all currently-living humans. (Though there's still a question of what to do with the remaining resources -- e.g., creating new humans or other new minds to live cool lives.)

A summary could be nice, maybe i'll try to write one if no one else does.

People summarizing this sounds great to me! Among other things, it's a good way to check whether the post miscommunicated in some way. :)