When you browse online for a new pair of shoes, pick a movie to stream on Netflix or apply for a car loan, an algorithm likely has its word to say on the outcome.
The complex mathematical formulas are playing a growing role in all walks of life: from detecting skin cancers to suggesting new Facebook friends, deciding who gets a job, how police resources are deployed, who gets insurance at what cost, or who is on a “no-fly” list.
Algorithms are being used — experimentally — to write news articles from raw data, while Donald Trump’s presidential campaign was helped by behavioural marketers who used an algorithm to locate the highest concentrations of “persuadable voters.”
But while such automated tools can inject a measure of objectivity into erstwhile subjective decisions, fears are rising over the lack of transparency algorithms can entail, with pressure growing to apply standards of ethics or “accountability.”
Data scientist Cathy O’Neil cautions about “blindly trusting” formulas to determine a fair outcome.
“Algorithms are not inherently fair because the person who builds the model defines success,” she said.
O’Neil argues that while some algorithms may be helpful, others can be nefarious. In her 2016 book, “Weapons of Math Destruction,” she cites some troubling examples in the United States:
– Public schools in Washington DC in 2010 fired more than 200 teachers — including several well-respected instructors — based on scores in an algorithmic formula which evaluated performance.
– A man diagnosed with bipolar disorder was rejected for employment at seven major retailers after a third-party “personality” test deemed him a high risk based on its algorithmic classification.
– Many jurisdictions are using “predictive policing” to shift resources to likely “hot spots.” O’Neill says that depending on how data is fed into the system, this could lead to the discovery of minor crimes and a “feedback loop” which stigmatises poor communities.
– Some courts rely on computer-ranked formulas to determine jail sentences and parole, which may discriminate against minorities by taking into account “risk” factors such as their neighbourhoods and friend or family links to crime.
– In the world of finance, brokers “scrape” data from online and other sources in new ways to make decisions on credit or insurance. This too often amplifies prejudice against the disadvantaged, O’Neil argues.
Her findings were echoed in a White House report last year warning that algorithmic systems “are not infallible — they rely on the imperfect inputs, logic, probability, and people who design them.”
The report noted that data systems can ideally help weed out human bias but warned against algorithms “systematically disadvantaging certain groups.”
Zeynep Tufekci, a University of North Carolina professor who studies technology and society, said automated decisions are often based on data collected about people, sometimes without their knowledge.
“These computational systems can infer all sorts of things about you from your digital crumbs,” Tufekci said in a recent TED lecture.
“They can infer your sexual orientation, your personality traits, your political leanings. They have predictive power with high levels of accuracy.”
Such insights may be useful in certain contexts — such as helping medical professionals diagnose postpartum depression — but unfair in others, she said.
Part of the problem, she said, stems from asking computers to answer questions that have no single right answer.
“They are subjective, open-ended and value-laden questions, asking who should the company hire, which update from which friend should you be shown, which convict is more likely to re-offend.”
Frank Pasquale, a University of Maryland law professor and author of “The Black Box Society: The Secret Algorithms That Control Money and Information,” shares the same concerns.
He suggests one-way remedy unfair effects may be to enforce existing laws on consumer protection or deceptive practices.
Pasquale points at the European Union’s data protection law, set from next year to create a “right of explanation” when consumers are impacted by an algorithmic decision, as a model that could be expanded.
This would “either force transparency or it will stop algorithms from being used in certain contexts,” he said.
Alethea Lange, a policy analyst at the Center for Democracy and Technology, said the EU plan “sounds good” but “is really burdensome” and risked proving unworkable in practice.
She believes education and discussion may be more important than enforcement in developing fairer algorithms.
Lange said her organisation worked with Facebook, for example, to modify a much-criticized formula that allowed advertisers to use “ethnic affinity” in their targeting.
Others meanwhile caution that algorithms should not be made a scapegoat for societal ills.
“People get angry and they are looking for something to blame,” said Daniel Castro, vice president of the Information Technology and Innovation Foundation.
“We are concerned about bias, accountability and ethical decisions but those exist whether you are using algorithms or not.”