Update: Reason on the discussion.
Why is that a threat?
No one knows what will happen when computers become smarter than their creators. Computer power has doubled every 18 months since 1956, and some AI experts believe that in the next century, computers will become smart enough to understand their own designs and improve upon them exponentially. The resulting intelligence gap between machines and people, Bostrom said, would be akin to the one between humans and insects. Computer superintelligence could be a boon for the human race, curing diseases like cancer and AIDS, solving problems that overwhelm humans, and performing work that would create new wealth and provide more leisure time. But superintelligence could also be a curse.
What could go wrong?
Computers are designed to solve problems as efficiently as possible. The difficulty occurs when imperfect humans are factored into their equations. “Suppose we have an AI whose only goal is to make as many paper clips as possible,” Bostrom said. That thinking machine might rationally decide that wiping out humanity will help it achieve that goal — because humans are the only ones who could switch the machine off, thereby jeopardizing its paper-clip-making mission. In a hyperconnected world, superintelligent computers would have many ways to kill humans. They could knock out the internet-connected electricity grid, poison the water supply, cause havoc at nuclear power plants, or seize command of the military’s remote-controlled drone aircraft or nuclear missiles. Inventor Elon Musk recently warned that “we need to be super careful with AI,” calling it “potentially more dangerous than nukes.”
Meh. Not too worried. The software to design and create AI is made by humans. Just think about how often the software you are using to read this, has to be debugged, fixed, patched, and updated.