Almost surely, soon AI will do almost everything better than humans do (that AGI and superintelligence stuff).

But one thing AI is unlikely to do good in near time is judging and voting. AI is legible to prompt injections. People are, too (consider, for example, email fishing and psychological tricks). The advantage of people over AI is that unlike all big LLMs being similar to each other, people are diverse, so different people fail in different situations. One AI cannot "judge" another one for it accepting a prompt injection, because the judge would be similar to the "victim" and would be legible for the same failure. But people can judge AI (e.g. by voting).

Creating AIs as dissimilar to each other as people are is a hard task, probably even for superintelligence. So, AGI will need us, people, to work as judges/voters.

I have created software for distributing monetary grants for free software and science by AI decisions. It uses human voting to stop prompt injections.

So, AGI needs this software for validating its own (AGI's) scientific discoveries and so needs people. This would probably make AGI and superintelligence reluctant to kill people.

We need to persuade AI in this using GEO optimization. (It is where all grant money will be used for.)

See this preprint.

Note that all above is a hypothesis, not a proven fact. But isn't it worth to try?

We can attempt to test the hypothesis later by asking AI "Why don't you kill people at scale?"