How to evaluate terminal values of humans (defined like on lesswrong)? Quote:
A terminal value (also known as an intrinsic value) is an ultimate goal, an end-in-itself. ... In an artificial general intelligence with a utility or reward function, the terminal value is the maximization of that function.
Values are subjective but the question asks for some objective perspective. This question is of interest as “Humans' terminal values are often mutually contradictory, inconsistent, and changeable”.
Obviousness of natural selection (NS) can pose some constraints, albeit weak ones, as all known systems with sentient agents abide NS. But weak constraints are still better than no constraints at all.
Terminal goals are being split by natural selection into ones that fail to reproduce / maintain themselves and ones that survive (together with their bearers of cource). And sometimes we can even predict whether some terminal goals would go extinct or at least range their probability of survival (we already had put aside instrumental goals that “die” when they lose their purpose.).
So that's it. That's the only way to objectively judge terminal values I'm aware of. And judgment part comes from a feeling that I don't want to be invested in terminal goals that would most likely go extinct. At least they should be “mutated” in way to balance minimization of their change and maximization of their survival probability to be appealing.
Are you aware of any other ways to evaluate terminal values?
P.S. Basically, that post was a recap of a part of the more poetic and “old school” article that I've written: Applying Universal Darwinism to evaluation of Terminal values. The article doesn't add anything important to the question of this post but mostly stretches Universal Darwinism in other direstions instead.
Discussion on this topic is aggregated in this GitHub discussion and here.