An international team of scientists from the United States, Australia, Taiwan, and Canada has conducted studies, published in The Lancet Digital Health, that report AI can predict a person’s race with 90% accuracy using X-rays and CT scans. Problem existing: scientists have no clue how it happens…

“When my graduate students showed me some of the results that were in this paper, I actually thought it must be a mistake. I honestly thought my students were crazy.”

Marzyeh Ghassemi, an MIT assistant professor of electrical engineering and computer science, told the Boston Globe.

The first signs of discovery were that the scientists noticed AI programs for examining chest X-rays would miss signs of illness in Black patients more so than in other races. Some of the team even questioned, “How can that be if computers cannot tell the race of a person?”

The AI was then taught by researchers using large numbers of race-labeled images of different parts of the body. There were no obvious markers of race in the images. The AI then identified race in the unmarked images with 90% accuracy and could also differentiate Black patients from white even when images were from people of the same size, age, and gender.

Even though the discovery is great for medical innovation, it also brings up the question surrounding AI-based diagnostic systems and how they might unintentionally generate racially biased results.

The initial belief, given by Ghassemi, is that it is all related to melanin. X-rays and CT scanners detect the higher melanin content of darker skin and embed this information in the digital image in some way gone unnoticed. Another belief is from Alan Goodman, a professor of biological anthropology at Hampshire College, suggesting that the AI is actually picking up differences resulting from geography. However, osteoarcheologists and geneticists have found no evidence of major racial differences in the human genome, but they do find major differences between people based on where their ancestors lived.

Just by studying oxygen, strontium, and sulfur from human bones, scientists can discover where a person was born and raised and can even find evidence of diseases such as trauma, leprosy, and syphilis. Another finding using the same isotopes from human bones is a person’s living conditions and lifestyle.

“Instead of using race, if they looked at somebody’s geographic coordinates, would the machine do just as well? My sense is the machine would do just as well.”

Alan Goodman said.