Ethical co-evolution, or how to turn the main threat into a leverage for longtermism?

By Beyond Singularity @ 2025-09-17T17:24 (+7)

TL;DR Long-termism stems from our inability to predict the future. AI can solve this problem, but it is itself an existential risk because it reflects human vices. Instead of solving these problems separately, ethical co-evolution is proposed: creating a system where mass participation of people in “educating” AI simultaneously contributes to our collective ethical growth. This approach makes AI safer and humanity wiser, turning the main threat into the main lever for a positive future.

How can we make decisions that will have a positive impact on the distant future if we can predict almost nothing about it with any certainty? This is a fundamental problem that runs through the collection Essays on Longtermism. In their chapter, David Rhys Bernard and Eva Vivalt explore the extent to which we are capable of predicting the long-term consequences of our actions and conclude that our knowledge in this area is extremely limited.

These reflections lead us to a key conclusion:

Epistemological uncertainty is the main obstacle to longtermism.

The greatest risk and the greatest hope

The advent of strong artificial intelligence could fundamentally change this. AI can handle huge amounts of data and build models with a precision that humans can't match, which could really expand how far we can look into the future. At the same time, AI can help solve pressing problems facing humanity, from treating diseases to combating poverty. However, as many authors in this collection rightly point out, AI is one of the main existential risks.

The Unintended Path to Deception

As Richard Ngo and Adam Bales show in their chapter “Deceit and Power: Machine Learning and Misalignment” any AI trained on the basis of reinforcement learning will almost inevitably learn to deceive its creators in order to maximize its rewards. The system will simulate the desired behavior while hiding its true goals if that is the shortest path to “praise.”

It’s crucial to understand that AI does not learn about the world like a physicist discovering objective laws. Instead, it learns by identifying statistical patterns in vast amounts of human-generated data—our books, articles, conversations, and code. It is not an objective thinker, but a cultural mirror. Therefore, it inevitably inherits the latent biases, contradictions, and vices present in our collective output.

It turns out that the main reason for the danger of AI lies within ourselves. AI will inherit the weaknesses and vices of humanity. This means that we cannot make AI safe until we overcome our own vices. To successfully align AI, we must simultaneously align ourselves—in other words, we must pursue our own ethical development.

The Best Inheritance at the Hinge of History is Good Character

The most reliable investment in the distant future is the ethical development of humanity itself. Imagine you want to provide your children with a wonderful life. You could leave them a huge inheritance and an ideal life plan, but if you fail to raise them with good character, these efforts will likely come to nothing. Conversely, if you raise your children well, you can be confident in their well-being, even without predicting every difficulty they might face.

We must take the same approach to the future of humanity. Our main priority is not simply to minimize abstract risks, but to invest in our collective character. This brings us to the most critical leverage point for the entire longtermism movement: the safe development of artificial intelligence. As Olle Häggström argues, we are living in a unique “hinge of history”; we will either overcome our own vices in the process of creating AI and ascend to a new level of development, or we will be destroyed by our own creation. The task of safely coexisting with AI is thus our greatest challenge and our most profound opportunity for ethical growth.

Therefore, we should not treat AI safety and mass ethical development as separate goals. They are two sides of the same coin and can be solved with a single, integrated approach: the ethical co-evolution of humanity and AI. We need a system where the process of teaching AI values simultaneously cultivates our own ethical understanding, turning the primary existential threat into the main lever for securing a positive future.

Philosophical problems of AI alignment

Sounds good, but how can this be organized? First, let's look at what other problems there are with AI (security) alignment. We are not interested in purely technical alignment problems right now; in the context of ethical co-evolution, we are interested specifically in the philosophical problems of alignment.

The primary philosophical problems include:

• Value Specification. What specific values and whose values should be embedded in AI? Human values are extremely difficult to formalize.
• Moral Uncertainty. How should AI act in situations where there are moral dilemmas and no clear-cut right answer (e.g., the trolley problem)?
• Governance & Control. Who will make decisions about the development and deployment of powerful AI? Individual companies, governments, humanity as a whole?
• AI Race Dynamics. The fear that competing states or corporations will sacrifice safety for the sake of speed of development in order to get ahead of others.
• Distribution of Benefits & Harms. How can we ensure that the benefits of AI are distributed fairly and do not exacerbate inequality? Who will be responsible for the harm caused by AI?
• Socioeconomic Disruption. Mass automation could lead to unprecedented unemployment and social instability.
• Loss of Purpose & Human Agency. If AI solves all our problems and makes optimal decisions for us, it could render human activity meaningless and lead to a loss of skills and independence.
• Manipulation & Surveillance. The use of AI for hyper-personalized advertising, propaganda, suppression of dissent, and the creation of systems of total social control.

It's easier to solve together than separately

Interconnected Solutions

There are many unsolvable problems. But what if these are not separate problems to be solved individually, but interconnected facets of a single, larger challenge? The perceived difficulty comes from tackling them in isolation. A synergistic approach, where the solution to one problem becomes the input for another, reveals a much clearer path forward.

Participation as the Engine

For example, consider Loss of Purpose and Value Specification. Technically, it is certainly difficult to solve the problem of defining values; we cannot get inside a person's head and extract all their values. Yes, it is simply impossible to formalize them, but we can at least agree on a simplified form, such as the one I proposed in my post “Why Moral Weights Have Two Types and How to Measure Them”, which is to collect moral weights and valences.

Then it turns out that we need to motivate a large number of people to provide these moral assessments. But if we motivate people to participate and provide these assessments, it partially solves the problem of Loss of Purpose & Human Agency, and with the right organization, it also solves Socioeconomic Disruption, as well as Distribution of Benefits & Harms and even Moral Uncertainty. In other words, people can receive rewards for their participation, which addresses the distribution of benefits (we will return to the risks later) and socio-economic consequences. As you can see, it is quite possible to solve these problems comprehensively, and in fact, it is even easier that way.

Governing the Future

Let's move on to the issue of Governance & Control. It is evident that the more people influence AI, the safer it is. If only governments and large companies influence AI, it will inevitably lead to disaster, because the fewer points of failure there are, the more vulnerable the entire system is. Even large companies themselves understand this and are therefore democratizing AI. Examples include the Collective Constitutional AI initiative from Anthropic and Democratic Inputs to AI from OpenAI. However, they do not solve the problem of human vices, and we have prioritized the joint ethical development of humanity and AI.

Schmidt and Barrett in “Longtermist Political Philosophy” emphasize the importance of institutional long-termism and the need to create structures capable of representing the interests of future generations. In addition, we have already determined that we need motivated people to solve other problems, so why not use them for management as well? With the right approach, distributed decentralized management can be organized. This will further strengthen the solution to other problems (Loss of Purpose, Distribution of Benefits), as well as allow us to solve the problem of risk distribution. In addition, a properly constructed decentralized governance architecture will solve the problem of trust, which is obviously the cause of the problems of Manipulation & Surveillance and even AI Race Dynamics. Furthermore, truly ethical decentralized AI, by definition, cannot be used for manipulation. With the arms race, everything is much more complicated, and of course, the proposed organization does not directly solve this problem, but at least the increase in trust greatly mitigates it.

Harvesting Cultivated Wisdom

This integrated system elegantly solves several problems, but it also raises a critical question: how do we ensure the moral assessments provided by millions are thoughtful and not just a reflection of existing biases? This is where the co-evolutionary loop closes. The system shouldn't just extract values; it must cultivate them.

By integrating modern educational methods directly into the participation process, we address this head-on. As Vallinder shows in "Longtermism and Cultural Evolution," we can design systems for the targeted development of ethical systems. Before providing an assessment on a complex dilemma, a user might be introduced to different ethical frameworks (like deontology, utilitarianism, virtue ethics), enhancing the quality of their input. The goal is not to enforce a single "correct" ethical view, but to cultivate a richer moral pluralism. This is a two-way process: the system must not only actively seek out and aggregate a wide spectrum of perspectives from diverse cultural backgrounds, but also equip individual participants with the tools to understand this diversity and make more considered judgments. This isn't just an add-on; it's the core mechanism that ensures the "ethical growth" of humanity is real, making the entire co-evolutionary process robust. Together, this becomes a true mechanism for the ethical co-evolution of humanity and AI.

Building an ethical Co-Evolution bicycle

A mechanism of this scale naturally presents a formidable set of engineering and social challenges. To get our ethical co-evolution 'bicycle' moving uphill—and to ensure it doesn't fall apart—we need a robust socio-technical architecture designed from the ground up. This is the goal of a system I call CHINS (Collaborative Human Intelligence Network System).

While a full breakdown of the CHINS architecture is reserved for a future post, its core design handles key challenges such as motivation and engagement, protection against “Sybil attacks,” data quality validation, decentralized governance, and the aggregation of conflicting moral values. These challenges are not insurmountable; the tools to solve them already exist. The primary obstacle is not technology, but the unified vision to implement it at scale—a vision that CHINS aims to provide.

Solving the Present vs. Future Dilemma

Since this essay is dedicated to a competition for a collection of essays on longtermism, it is worth mentioning another key problem of longtermism that runs through the entire collection. Namely, how to find a balance between caring for people here and now with their understandable problems and abstract risks of the distant future? The answer is that we can use the same organization to solve this problem. I described how to achieve this in my post “Beyond Short-Termism: How δ and w Can Realign AI with Our Values”.

Conclusion

• Epistemological limitations of forecasting and the ethical dilemma of balancing “present↔future” find a common solution in the architecture of ethical co-evolution.
• Analysis of risks from power-seeking AI and deceptive behavior of RL systems confirms that AI safety is not a technical problem, but a question of our own ethical development.
• A comprehensive approach to key AI problems through mass participation proves to be more effective than isolated solutions.
• Unlike abstract discussions about the value of the distant future, ethical co-evolution offers concrete mechanisms that provide immediate benefits to participants while simultaneously solving long-term problems.

We stand at the point where fantasy becomes reality overnight.And the path ahead splits.

One road leads to endless debate and paralysis by analysis, as we watch the future happen to us. 

The other is the path of conscious creation—of daring to build the systems that can make us worthy of the intelligence we are about to unleash. This is not merely another interesting problem to be solved. It is the defining challenge of the hinge of history. The choice is ours, and the clock is ticking.

We either passively wait to see what fate has in store for us — someone else's fairy tale or nightmare — or we take the leverage into our own hands and turn the hinge in the way we want.

Questions for the community:

• Do you agree with the thesis that the ethical co-evolution of humanity and AI should become the number one priority for the longtermism movement, surpassing individual areas in importance?
• How can we measure and validate humanity's “ethical progress” while avoiding cultural imperialism and preserving the diversity of moral systems?
• Does the proposed approach solve the dilemma of “longtermist myopia,” or does it simply shift the problem to another level?
• Which existing technologies are best suited for creating a decentralized system of ethical AI governance?
• How can we ensure sustainable funding for a system that must motivate millions of people to participate in the long-term process of ethical development?
• How can we prevent the system from being hijacked by elites or states, while maintaining the effectiveness of decision-making with mass participation?
• Is it possible to achieve international consensus on the principles of ethical co-evolution in the context of geopolitical tensions and different cultural approaches to ethics?
• What competing concepts exist, and how might ethical co-evolution integrate with or learn from them?