Creating smarter, more accurate systems requires a hybrid human-machine approach, according to researchers at the University of California, Irvine. In a study published this month in Proceedings of the National Academy of Sciences, they present a new mathematical model that can improve performance by combining human and algorithmic predictions and confidence scores.
Humans and machine algorithms have complementary strengths and weaknesses. Each uses different sources of information and strategies to make predictions and decisions. We show through empirical demonstrations as well as theoretical analyses that humans can improve the predictions of AI even when human accuracy is somewhat below [that of] the AI—and vice versa. And this accuracy is higher than combining predictions from two individuals or two AI algorithms.
– Mark Steyvers, Co-author
To test the framework, researchers conducted an image classification experiment in which human participants and computer algorithms worked separately to correctly identify distorted pictures of animals and everyday items—chairs, bottles, bicycles, trucks. The human participants ranked their confidence in the accuracy of each image identification as low, medium or high, while the machine classifier generated a continuous score. The results showed large differences in confidence between humans and AI algorithms across images.
When predictions and confidence scores from both were combined using the researchers’ new Bayesian framework, the hybrid model led to better performance than either human or machine predictions achieved alone. While past research has demonstrated the benefits of combining machine predictions or combining human predictions—the so-called ‘wisdom of the crowds’ – this work forges a new direction in demonstrating the potential of combining human and machine predictions, pointing to new and improved approaches to human-AI collaboration.
This interdisciplinary project was facilitated by the Irvine Initiative in AI, Law, and Society. The convergence of cognitive sciences—which are focused on understanding how humans think and behave—with computer science—in which technologies are produced—will provide further insight into how humans and machines can collaborate to build more accurate artificially intelligent systems.
To merge several decisions into one, researchers weighted individual responses by decision confidence – the algorithm’s self-estimated confidence, and the measurements from the humans’ brain readings, transformed with a machine-learning algorithm. We found that an average group of just humans, regardless of how large the group was, did better than the average human alone – but was less accurate than the algorithm alone. However, groups that included at least five people and the algorithm were statistically significantly better than humans or machines alone.
Pairing people with computers is getting easier. Accurate computer vision and image processing software programs are common in airports and other situations. Costs are dropping for consumer systems that read brain activity, and they provide reliable data. Working together can also help address concerns about the ethics and bias of algorithmic decisions, as well as legal questions about accountability.
As reported by OpenGov Asia, U.S. researchers have also developed an AI tool, called CitizenHelper to provide insights into online behaviour. This tool can sort through millions of tweets to identify behaviours that could assist emergency agencies and give them an understanding of the population’s attitudes.
The tool helps these CERTs to scale work that would be difficult for humans to do alone. The head of the research team says that humans are good at contextual understanding to filter content but they cannot scale. Machines, on the other hand, are good at scaling, but they do not deeply understand the context very well. Hence, a human-AI teaming approach is invaluable. The algorithms need humans to help them improve their accuracy. CitizenHelper allows this very seamless interactive mechanism for humans and computers. The humans can provide feedback to the machine on what the machine has predicted.
