Terminology: Black box
AI is a very technical field, and having a career in AI today requires postgraduate training in computer science and some advanced mathematics. The good news is that, in order to have an intelligent discussion about evil AI, you do not need to know anything about how AI actually works ‘under the hood’. My computer science colleagues may not like me saying this. But we can go a long way towards understanding machine misbehavior without any computer science knowledge whatsoever. We just need to know what the machines do.
Perhaps an analogy is helpful. We still do not fully understand the human brain. Yet, for thousands of years, we have been able to regulate human behavior using laws, social norms, etc. For the vast majority of human history, the brain was essentially a blackbox to us. And in many ways it still is. Yet we have a good understanding of the causes and remedies for anti-social behavior like bullying, or criminal behavior like tax avoidance or physical violence.
There are many reasons we may not have access to the AI black box. For starters, many AI algorithms are proprietary technologies protected by intellectual property law and industrial secrecy. Second, in many cases, even the computer scientists who built the algorithms do not fully understand the behavior of their own creations. This is because AI systems can essentially program themselves, by learning from data or their own experiences. And these learned programs can sometimes defy interpretability. In these cases, our attention must be on observable ‘machine behavior’.
Note that embracing AIs as black boxes does not mean that we should not seek transparency about the inner workings of machines, whenever possible and necessary. It’s just that there is a difference between transparency of thought, and transparency of behavior. We cannot read the minds of politicians, but we can demand that they are transparent about who they associate with, where the money goes and so on. Similarly, we do not have a full understanding of the brain of human drivers, yet we issue driving licenses based on well-defined tests, and we ensure people continue to obey traffic rules. We focus on the human’s driving behavior, and it happens to be sufficient for the purpose of regulating the activity of driving. The same can, in principle, be done for self-driving cars, and for AI behavior more generally.
References
Rahwan, I. et al. Machine behaviour. Nature 568, 477–486 (2019).
Rodriguez, J. What is Machine Behavior? KDnuggets. (2019)