He grew so frustrated with the fact that he was allowed to spend 20 times as much on his card that he posted an expletive-laced series of Tweets describing Apple Card as “such a ——- sexist program”.
The couple, he pointed out, “file joint tax returns”, split their property and have been “married a long time”. So why was it that her credit limit was so vastly lower than his?
“Computer says no”? That was his allegation. Except these days the phrase needs updating to “Algorithm says no” – algorithms being the rules and instructions by which computers make their calculations.
His ire was heightened by the fact that when he and his wife signed up for a credit checking service they discovered that, “HER CREDIT SCORE WAS HIGHER THAN MINE!!!”
When she called Apple to complain, she was told. “I swear we’re not discriminating, ‘IT’S JUST THE ALGORITHM’. I —- you not. ‘IT’S JUST THE ALGORITHM!'”
The outburst had immediate effects. First, and most predictably, Jamie Heinemeier Hansson had her credit limit bumped up. But then the New York State Department of Financial Services (NYDFS) weighed in. “Financial services companies are responsible for ensuring the algorithms they use do not even unintentionally discriminate against protected groups,” replied NYDFS Superintendent Linda Lacewell, promising to “look into it”.
‘It’s big tech in 2019’
Humiliatingly for Apple, its revered co-founder Steve Wozniak then revealed that exactly the same thing had happened to him. His wife’s credit limit was a tenth of his, despite the fact that “we have no separate bank or credit card accounts or any separate assets. Hard to get to a human for a correction though. It’s big tech in 2019”.
Goldman Sachs, Apple’s partner in launching the credit card, released a statement insisting that applications are “evaluated independently”.
Its explanation of events contained two elements. The first noted that “We look at an individual’s income and creditworthiness, which includes factors like personal credit scores, how much debt you have and how that debt has been managed.”
The second noted that: “In all cases we have not, and will not, make decisions based on gender.”
Amazon’s ‘algorithmic bias’
In other words data, not prejudice, was the basis of their decision. All of which sounds very fair. The problem is that while we instinctively think of computers as impartial calculating machines, prejudice can all too often be baked into the data they rely on.
Amazon knows. The e-commerce giant has long forged and secured its dominance by investing in automation. So it was no surprise when, in 2014, it began working on computer processes that would automate the task of selecting the best people for top jobs from a pile of CVs.
“Everyone wanted this holy grail,” a company insider working on the project told Reuters last year. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”
Unfortunately, the computers were trained on CVs submitted over 10 years. And because the tech industry is dominated by men, the vast majority of those CVs came from men, and the system learned to downgrade women. The programs began to look out for telltale phrases like “women’s chess club captain” and mark those candidates for the scrapheap.
It was a classic, self-reinforcing, case of what is known as “algorithmic bias”. It’s not intentional. But computers, told to learn from unfair history, echo and extend that history. Impartial? Not so much.
It’s all the more pernicious because of our own, human bias which somehow credits computers with all-seeing neutrality. Far from it.
The Amazon case is particularly interesting, because the company took active steps to correct the problem by insisting that the computers gave no weight to gender-related terms. But by the start of 2017, executives had reportedly lost faith in their capacity to fix the issue, in part because they could not guarantee that the computers would not find other ways to discriminate against candidates.
‘Black box’ problem
‘Simple algorithms, like simple instructions’, are easy to understand. But as artificial intelligence develops, algorithms become increasingly complex, and ultimately it can become impossible to pinpoint why a computer has evaluated data in one way or another. Its decision-making process, in effect, becomes a “black box” – a mystery, both to those who designed it, and those affected by it.
That is why “explainable AI” is such a fashionable phrase. What’s the point of vastly sophisticated algorithms, critics argue, if it is impossible to justify their reasoning. “Show your working”, teachers once insisted to children sitting maths exams. Should computers not be subject to the same basic rule?
David Heinemeier Hansson certainly thinks so. “Apple’s black box algorithm thinks I deserve 20x the limit she does. No appeals work.”
Nonetheless, proprietary algorithms are used in the justice system. Most famous is the Correctional offender management profiling for alternative sanctions (Compas) in America, which claims to calculate the risk of reoffending and so help judges in sentencing. A study found that it was no more accurate at doing so than strangers recruited from the internet.
What is the solution? Because if there is consensus about anything, it is that automated, data-driven assessment of human behaviour is not going to go away. Quite the reverse.
The usual touted cure-all is greater diversity. If data sets were less skewed, this argument runs, then algorithms would not learn bias. But it is hard to correct what Caroline Criado Perez, in her award winning book Invisible women: Exposing data bias in a world designed for men calls a statistical “silence” about half of humanity.
Fixing the past is too big a task. Rather, it is incumbent upon design teams, whether of sports cars or payment cards, to fix the future by ensuring they are representative of the world they claim to be serving. The alternative, as Criado-Perez writes, is “a super-rational world, increasingly run by super-impartial supercomputers, [where] women are still very much De Beauvoir’s Second Sex – and that the dangers of being relegated to, at best, a sub-type of men, are as real as they have ever been.”
The Daily Telegraph London