Ok thanks for explaining your point of view. That really does help. I think there is a tendency to produce circular definitions in ethics. The classical ethical rules even seemed to befall that. Do unto others… well but what should I do? Here is something that would work if it were a good definition: only do things that will result in good. But wait, that’s circular. Same problem with agreeableness. That’s why we generally try to reduce “good” to other things in philosophy. The most popular, well known instance of that is “maximizing utility”, which is also an AI agent design. But what is the utility then? That is still circular. You see even the best philosophers haven’t completely solved this problem. You can find that by searching for “Peter Singer utility monster”.
That’s why in my own philosophical research, I opted for “universal meta-goals” while the AI doomsayer folks at FHI opted for “human preferences”. Human preferences do not explain what is good, it just models what humans prefer, that’s not necessarily good. However, I found at least one rule that is good, which is preserving and pervading life and information. Note that this is much different than agreeability or utility which still depend on some hypothetical subjective point of view (just as you explained an omnipotent god’s point of view, which can be a good story but not related to reality). Also, people don’t want things because they find their goals agreeable. People have many instincts and motivations genetically. If the subjective point of view is required to decide what is good, then the ethical theory is probably incomplete. At least, that’s what utilitarianism feels like, it’s as hollow as other forms of neoliberal nonsense. What other people think we should behave like is just the social norm, or the superego, it’s not necessarily good. In most religious societies, you’re expected to belittle women, but with a bit of thinking, we can infer that’s wrong.
That’s why most utilitarians try to reduce good to “good feelings”, i.e. stuff that makes people happy. So that’s a neurological state. But that’s still subjective and error-prone, so it’s not good enough and it gives no meaningful means of qualitative comparison.
So in fact I’m referring to a whole set of problems that philosophers of ethics haven’t even realized exist, at least since ancient times. You may find the paper enjoyable. But it might be hard to read if you aren’t familiar with what it refers to, I’m still sure you’ll find it interesting.
There are many other papers in AGI literature as I said. Here is another reference that summarizes what I think of as the worse approach to AI ethics, which is something like imagining badly programmed reinforcement learning agents. This is still something important to know, but if you read carefully Tom Everitt concedes that AI researchers do not know and do not agree on the right theory of morality. That’s important, we don’t really know it.
That’s why, some researchers like me are looking for Cosmopolitan values. If you don’t have a great theory of morality and ethical behavior in general, no you can’t reduce it to people’s desires, judgement or knowledge. Because AI will be a lot more than people.
That’s more in line with ancient greek philosophy than all this other stuff that looks like behavioral / evolutionary psychology (which looks mostly like pseudoscience to me, I’m afraid).