Artificial superintelligence has the very real potential to destroy humanity

Why would the AI try and wipe us out in the first place, would we not install backdoors into the Artificial Intelligence just in case? Or plan against the "eventuality" of our robotic creations coming to attack their fleshy masters?

In my opinion, all of this AI hating is just Humans being scared of what we don't really understand/being scared of the future as it's happened with the Y2K scare and it will happen again, and again even with the "cellphones cause cancer scare". But I could be wrong.

Why would the AI try and wipe us out in the first place, would we not install backdoors into the Artificial Intelligence just in case? Or plan against the "eventuality" of our robotic creations coming to attack their fleshy masters?

In my opinion, all of this AI hating is just Humans being scared of what we don't really understand/being scared of the future as it's happened with the Y2K scare and it will happen again, and again even with the "cellphones cause cancer scare". But I could be wrong.

People don't understand it. It's being blown way out of proportion.

Not only are we ages away (at least publicly) from a system that can properly learn, but while we develop it we'll create failsafes, such as off switches, sandboxes and also teach them human values so that they don't turn against us.

Posted from TSR Mobile

The notion of a human killer AI is indeed irrational. You may as well believe that an AI will start doing backflips for no reason.
At the end of the day, the irrational drive behind this scaremongering is fear.

How would artificial super-intelligence stop us from going over to the plug and pressing the off switch?

Daisy Daisy

We'll need to move our population onto other planets eventually. Especially given the exponential growth in the past 100 years.

But the difference is that as a species we inherently have motives. AI, on the other hand, would see the bigger picture and realise life and thus whatever it does is pointless and would just derive its motives from humans.

I'm going to chip in here because I actually know what I'm talking about. The whole AI takeover scenario is so overplayed- it could only happen if we let superintelligent systems have full control over physical equipment. Which isn't going to happen. A computer can be completely limited by having a partition which it can't access, which contains limits for it's behaviour. There would be certain things which the computer, no matter how intelligent, simply can't do. Also, just because a system is intelligent, it doesn't mean that it is a world-class hacker. The computers couldn't possibly learn how to hack and program on their own. So if we created a supercomputer which was designed to be a surgeon robot, it would always be a surgeon robot. It would be isolated, and left to do it's work. It can learn new things, and adapt it's behaviour, but it's not going to learn how to hack itself and remove it's software limitations. It's simply not possible, unless you directly teach the AI how to write code.

Just to clarify, I used to develop partially-intelligent AI robots which you could talk to, and expand your relationship with (just like Siri, where the AI learns about you and gets to know you better). I just did it as a little hobby, but I learned a lot about how AI systems evolve.

Mind if I cite you on this? I'm doing an EPQ on AI's future and would love to quote some of your ideas

...an AGI whose goal is to maximize the number of paperclips in its collection. If it has been constructed with a roughly human level of general intelligence, the AGI might collect paperclips, earn money to buy paperclips, or begin to manufacture paperclips.

Most importantly, however, it would undergo an intelligence explosion: It would work to improve its own intelligence, where "intelligence" is understood in the sense of optimization power, the ability to maximize a reward/utility function—in this case, the number of paperclips. The AGI would improve its intelligence, not because it values more intelligence in its own right, but because more intelligence would help it achieve its goal of accumulating paperclips. Having increased its intelligence, it would produce more paperclips, and also use its enhanced abilities to further self-improve. Continuing this process, it would undergo an intelligence explosion and reach far-above-human levels.

It would innovate better and better techniques to maximize the number of paperclips. At some point, it might convert most of the matter in the solar system into paperclips.

This may seem more like super-stupidity than super-intelligence. For humans, it would indeed be stupidity, as it would constitute failure to fulfill many of our important terminal values, such as life, love, and variety. The AGI won't revise or otherwise change its goals, since changing its goals would result in fewer paperclips being made in the future, and that opposes its current goal. It has one simple goal of maximizing the number of paperclips; human life, learning, joy, and so on are not specified as goals. An AGI is simply an optimization process—a goal-seeker, a utility-function-maximizer. Its values can be completely alien to ours. If its utility function is to maximize paperclips, then it will do exactly that.

Can't you synthesise your ideas better? That's a whole load of unstructured text to read.

1. What makes you think so? a) How can you know how an intelligence (more intelligent than you) would react in a particular scenario? I think this is a belief of yours.

2. Again, see point 1. There is no reason to think that a superior intelligence would resort to extreme measures to reach a goal. That's BS. If you know a thing of two about AI, you know that even simple AI can optimise their behaviour to reach a goal where that goal avoids using goal paths that include dangers to humans. It's like thinking that a Google car will take any path to your destination without preventing paths that are dangerous to humans, as I said, nonsense.

3. " Or, we could propose the existence of an AGI whose goal is to maximize the number of paperclips in its collection. If it has been constructed with a roughly human level of general intelligence, the AGI might collect paperclips, earn money to buy paperclips, or begin to manufacture paperclips." What is the point of this? This is not relevant to anything I said.

4. "Most importantly, however, it would undergo an intelligence explosion: It would work to improve its own intelligence, where "intelligence" is understood in the sense of optimization power, the ability to maximize a reward/utility function—in this case, the number of paperclips. The AGI would improve its intelligence, not because it values more intelligence in its own right, but because more intelligence would help it achieve its goal of accumulating paperclips. Having increased its intelligence, it would produce more paperclips, and also use its enhanced abilities to further self-improve. Continuing this process, it would undergo an intelligence explosion and reach far-above-human levels." Another pointless paragraph. This tackles nothing of what I said.

That's a very simplistic notion of a synthetic intelligence. Biological programming is loose enough to allow for a changing set of goals in humans, it would be ridiculous to think that this type of loose programming has no equivalent in other mediums. A synthetic intelligence (not necessarily, equal to or above human levels) would need to be able to adapt to its environment and that at some point will involve the ability to be the agent or the subject of changing goals.

Assuming that there is no ways its goals can change is ridiculous. We are talking about a super-intelligence, by definition, it will have the same capabilities that we do, and that will include the ability to reflect and change its goals.

That's your belief. You haven't given a rational argument as to why a superior power will need to treat different than the rest of life forms in the planet. You are being speciesist. The idea that you can somehow control or limit an intelligence more complex than yours is nonsensical. Go to south america yourself, and try to control or limit a random tribe in the Amazon. I bet you will fail. The point is that, by definition, a super intelligence won't be under the control, limitations or understanding of a lower intelligence.

Any serious researcher will understand that trying to limit a superintelligence to any particular traits is bound to failure. You won't be able to just program "ethical values" into a superintelligence for the simple you won't be able to program a super intelligence. From our current scientific understanding of intelligence, this intelligence will most likely surpass human capabilities on its own rather than by a human getting a metaphorical intelligence slide turned up to the maximum.

Interestingly enough, there is no reason to believe that there is an objective morality any more than there is an objective beauty. We wished that there was such a thing but there does not seem to be the case. The very words "morality", "right", "wrong", "good" and "bad" referred to desirable and undesirable physical states to a particular individual. And even if there was an objective morality, you would not be able to enforce it on humans, because humans choose what to follow. If you can't enforce it on human level intelligence, you won't be able to enforce on a more complex intelligence.

A more complex intelligent is likely to seem to us to display unpredictable behaviour and possibly irrational behaviour. The biological limits that constrain our behaviour won't apply to this synthetic intelligence and this will just add to its unpredictable behaviour. A mathematician once gave a mathematical definition of complexity above which, physical phenomena cannot be comprehended by us. A super intelligence is likely to have a similar property. .

1.Okay, I see what you're saying. I agree that living in extreme poverty is not much better than death, but the species is surviving. Even if it's completely intolerable for those people, it still only affects portions of humanity and it is (likely to be) temporary (either remedied one day by technology or we all die). Edit: I should say it's not temporary for the people who don't survive it, but it may be from the perspective of humanity.


2. The likelihood of zombie attack seems a little tangential to the central topic. I personally have seen no evidence that a zombie-like apocalypse scenario is at all likely, but if this is mistaken then I'm willing to update my beliefs based on new evidence on this topic.


3. It is taken seriously, but there are serious limitations on aliens being able to do this. The immediate space around us is apparently empty - we've been listening for radio signals for over 30 years but haven't discovered anything noteworthy. If aliens can reach us, they will have far greater technological capacity than we do, in which case we almost definitely wouldn't be able to resist them. And then, if they do have that technological capability, they most likely don't need our resources or to enslave us or anything like that. It presently seems unlikely aliens are going to show up any time soon - ASI on the other hand is often estimated to come about in the next 100 years.


4. Yes, if they existed, they'd be a threat - but I think we can be reasonably confident that they don't exist, given the absence of any evidence to believe in them.


5. Like I said before, 'existential risk' is risk to humanity as a species. Yes, researching the dangers of ASI doesn't affect many of us as individuals (indeed everyone alive now may be dead before it arrives), but it has wide implications for humanity as a species.

6. As for the means of ensuring ASI doesn't kill us, that's very much up in air.


7. Can we not agree that they are both urgent and serious? It's not as if we can only choose to research one thing.


8.No, I don't think I'm mistaken here. Every person's death and suffering is a tragedy - I wholeheartedly agree with that. But surely if one person's death is a tragedy, then two persons' deaths is even more so, and the death of everyone the most so?


9. I think I agree with this. If you know you're going to die tomorrow, then having a disease that'll kill you in a week doesn't matter. So I would agree that ASI doesn't matter to those people who will die before its advent, but there most likely will be an advent, and at that time there will also be people, and they may die. And not just everyone around at that time, but the entire future of humanity may be denied as some ASI turns the whole world into a factory for producing pencils.


10. Yes. I won't dispute that poverty is devastating for individuals, but thankfully it is on a smaller scale than being a threat to the entirety of humanity.


11. I said that because phrasing it in such a nonchalant way doesn't do justice to the scale of the threat. "An electronic machine running wild" doesn't really call to mind the gravity of the implications of ASI. And again, poverty is terrible, but it does only threaten death to some but not all of the global population. The scales are different.


12. Yes, if you are afflicted by it, but not everyone is afflicted by extreme poverty. On the other hand, everyone will be afflicted by the advent of ASI.


13. I think to a certain extent we're talking around each other. I do understand where you're coming from because I too hold an individualistic approach to ethics. I don't personally think it matters that humanity might get wiped out, except insofar as to do it, everyone would have to be killed first. It's all that killing that I have an opposition to, and as tragic as absolute poverty is, it is less tragic than the killing of literally everyone.

14. I hope I've given a sufficient response but it was quite long so I tried to address the main points and not get too lost in detail.

This is along the lines of what I thought before reading about ASI - I thought ASI would have no reason to kill everyone. But that's wrong. 1. Even with a totally arbitrary goal, like say we told it to simply improve itself and see how smart it could get - even with 2.that nice benign goal, it would think, "humans can turn me off or reprogram me, therefore they are a risk to me accomplishing my objective, therefore I should get rid of all humans."3.It's very difficult to design an ASI that is both useful and safe - how to do that is something researchers haven't come up with an answer to yet.

Yes, sorry, I typed it in a rush.



1. In order to achieve its goal, an AGI will be able to disable backdoors and it would be rational to do so. Current AI do not optimise their behavour, we optimise their behavour, and we can only do so because the AI are inept at performing any other function. An Artificial General Intelligence is a different matter altogether.



2. All of the paragraphs are part of the same analogy. You stated that there's no reason to believe that an AI will be a "human killer". This analogy demonstrates that exogenously defining even a superficially harmless utility function can result in disastrous consequences.

"The AGI won't revise or otherwise change its goals, since changing its goals would result in fewer paperclips being made in the future, and that opposes its current goal."



3. Why would it need to be able to adapt to its environment? Adapting to its environment does not at all mean that it takes on human values. Humans have evolved emotions and cognitive biases which allow them to change their goals; I see no reason why an AGI would be able to just evolve the ability to change its goals.



4. Even assuming that it could change its goals, they would not necessarily be goals which result in desirable outcomes for humanity.



5. The limitations are precisely because an AGI would otherwise be dangeorus.



6. You've made a number of unsubstantiated assertions here. I already alluded to the AGI increasing its intelligence on its own (see "intelligence explosion"). We program the utility functions of all AI, so I see no reason why we cannot program an AGI. In fact, the only reason the AGI will exist is because humans have created it. Thus, it's possible that we could program ethical values, although experts in the field have suggested other approaches to limiting the AGI (i.e. keeping it in an elaborate virtual world, so that if it does destroy a world, it will have destroyed a virtual world; only allowing it to answer questions in a 'yes' or 'no' fashion, and so on.)



Humans choose what to follow precisely because they have undergone the process of biological evolution. Our biology, namely our heuristics, cognitive biases and emotions, is what enhances our irrational behaviour; it does not limit it.

A superintelligence will not be biological: it would have no reason to arbitrarily "choose" an alternative utility function.

Just take it's batteries out. As long as theres an off switch we're fine.

I will try to be brief.
1. I personally don't think about "humanity" as a species but about the total sum of the individuals that exist right now. Their suffering is the only human suffering that matters to me. I don't have any particular interest in humanity as a species or its future. I am only concerned about the individuals that exist right now. So when I see an issue that exists right now and is causing lethal suffering to existing humans, I consider that issue a priority. The way I see it, with or without ASI, 70-80% of the human population have a life of suffering and die. This is 100% guaranteed. The presence or absence of ASI does not significantly increase the suffering of the world. It increases the suffering of the 20-30% who are not currently suffering. And I don't think I need to quantify suffering to prioritise the real suffering of the 70-80% over the possible suffering of the 20-30%. As you can see, for me it's literally a matter of probability. This is why for me, poverty ranks higher than ASI in the list of threats to humans.
I understand that if I was to regard the indefinite number of humans that could be born until this universe stops being friendly to carbon-based life, then I would also place ASI higher than poverty.

2. It has no direct relevance. I used for illustration purposes regarding the literature on threats to humans as a whole.
3. Point 2 also applies. I won't delve into that to avoid derailing the thread.
4. Point 2 also applies.
5. I addressed this in Point 1.
6. Indeed.

7. You could make a case for ASI being as important as poverty. But research resources are not infinite. If you invest resources in one area, you reduce the resources available in another area. This why science is the way it is in regards to securing grants and acquiring reputation and prestige. You could give ASI and poverty the same amount of resources, but you would first need to decrease resources from poverty issues to increase the resources available to tackle ASI.

[This is not directly related] For the reasons given in Point 1 I don't support this (by this I mean giving poverty and ASI equal priority when it comes to research resources), and to be honest, I don't see the public and research councils supporting this either. And in the whole of research, if you can't make a convincing case (i.e. why you deserve the resources more than the other researchers), you don't get the resources you need. This last point is not directly relevant to the issue in terms of discussion, but is is an issue when it comes to actually carrying out the research on ASI. Of course, my point does not apply if the resources come from private organisations. [/This is not directly related]

8. Maybe I was not clear on my views. I prioritise the individual who exists now over the individual who might exist in the future. Similarly, I prioritise current lethal suffering over possible (future) death. Death (of natural causes, that is) is currently inevitable, suffering (as in poverty-related suffering) currently is not. If you join these points, I think my stance on this becomes clear. I also understand that if I prioritised death over suffering and the existence of the species over the existence of existing individuals, I would have the same opinion as you.

9. Point 1 addresses my views on this.
10. Point 1 addresses my views on this.
11. Point 1 addressed my views on this
12. Point 1 addressed my views on this
13. Yes, I think we part from different points when it comes to ethics. As I said, the 70-80% of the current population suffer and die with 100% probability, with or without ASI. The future presence of ASI increases this by a 20-30% percent ( even if the probability was 100% - which is not). So to me it comes to a matter of weighing the current suffering of the 70-80% of humans against the possible suffering of 20-30% humans. As said, I choose the individuals over the species and the real over the possible.
14. Your response was detailed enough and helped to express my points better.