How worried should I be about a childless Disneyland?

I don't think it matters that much (for the long-term) if the AI systems we build in the next century are conscious. What matters is how they think about what possible futures they can bring about.

If AI systems are aligned with us, but turned out not to be conscious or not very conscious, then they would continue this project of figuring out what is morally valuable and so bring about a world we'd regard as good (even though it likely contains very few minds that resemble either us or them).

If AI systems are conscious but not at all aligned with us, then why think that they would create conscious and flourishing successors?

So my view is that alignment is the main AI issue here (and reflecting well is the big non-AI issue), with questions about consciousness being in the giant bag of complex questions we should try to punt to tomorrow.

Several background variables give rise to worldviews/outlooks about how to make the transition to a world with AGIs go well. Answering this question requires assigning values to the background variables or placing weights on the various worldviews, and then thinking about how likely "Disneyland with no children" scenarios are under each worldview, by e.g. looking at how they solve philosophical problems (particularly deliberation) and how likely obvious vs non-obvious failures are.

That is to say, I think answering questions like this is pretty difficult, and I don't think there are any deep public analyses about it. I expect most EAs who don't specialize in AI alignment to do something on the order of "under MIRI's views the main difficulty is getting any sort of alignment, so this kind of failure mode isn't the main concern, at least until we've solved alignment; under Paul's views we will sort of have control over AI systems, at least in the beginning, so this kind of failure seems like one of the many things to be worried about; overall I'm not sure how much weight I place on each view, and don't know what to think so I'll just wait for the AI alignment field to produce more insights".

Do you mainly see these scenarios as likely because you don't think there is likely to be many beings in future worlds or because you think that the beings that exist in those future worlds are unlikely to be conscious?

I had some thoughts about the second case. I've done some research on consciousness, but I still feel quite lost when it comes to this type of question.

It definitely seems like some machine minds could be conscious (we are basically in existence proof of that), but I don't know how to think about if a specific architecture would be required. My intuition is that most intelligent architectures other than something like a lookup table would be conscious, but don't think that intuitions based on anything substantial.

By the way, there is a strange hard sci-fi horror novel called Blindsight that basically "argues" that the future belongs to nonconscious minds and this scenario is likely.