When evaluating AI, remember the baseline

If you read the news media today, you get the impression that the world is at the verge of catastrophic collapse. A 2016 YouGov survey asked people in 17 countries whether the world is getting better or worse. Only 11% of people believed the world to be getting better, while 58% thought it was getting strictly worse. In the US, 65% thought things were getting worse, and a mere 6% believed it was getting better.

Yet, when you consider objective indicators, the world is improving in many ways. There has been a massive, global alleviation of extreme poverty. Famine is all but eradicated. The global tree canopy increased by 2.24 million square kilometers between 1982 and 2016–an area larger than Alaska and Montana combined. And the chance of dying in a natural catastrophe has declined by nearly 99% since the 1920s and 1930s.

People have always lamented the “good old days.” Various psychological reasons account for this tendency. One example is what Harvard psychologist Daniel Gilbert calls ‘judgement creep’: When problems become rare, the goal post shifts, and we count more things as problems. But if we do not measure progress, we might fail to make informed decisions about tradeoffs.

Suppose we discover that an AI system for diagnosing a medical condition is 90% accurate for women, and 80% for men. We may wish to ban the system, due to its systemic discrimination. However, before we do so, we may want to consider the baseline. If the baseline, without the AI, is 70% accuracy for both men and women, then we know we are making a tradeoff between fairness (equal accuracy for men and women) and higher accuracy for everyone (20% and 10% improvement for women and men, respectively). Whether fairness or accuracy is more important is non-trivial. But without the baseline, we would be blind to the tradeoff, and only see the problems.

On the flip side, we should avoid being complacent about problems we have already solved. Often, solving one problem may create new problems. For example, by making transportation cheaper, self-driving cars may increase congestion and pollution. To improve the world with AI, we need good measurement of its impact in its entirety.

References

• Dinic, M. Is the world getting better or worse? YouGov https://yougov.co.uk/topics/politics/articles-reports/2016/01/08/fsafasf (2016).

• Bailey, R. & Tupy, M. L. Ten Global Trends Every Smart Person Should Know: And Many Others You Will Find Interesting. (Cato Institute, 2020).

• Levari, D. E. et al. Prevalence-induced concept change in human judgment. Science 360, 1465–1467 (2018).

• Crawford, K. The Atlas of AI. (Yale University Press, 2021).