An algorithm can be simplified as a set of instructions to the quickest and most efficient solution possible. For example, a recipe on how to make a pie can be considered an algorithm. Somebody, a long time ago, used trial and error to create the quickest and most efficient pie recipe.
In the article “The Black Box” we hear Cathy O’Neil speak about how algorithms can marginalize certain groups of people and can be biased against them. O’Neil is a mathematician and former Wall Street quantitative analyst. She has written a book titled “Weapons of Math Destruction,” where she talks about popular algorithms used by large companies and institutions often harm poor and working people and benefit elites. She says that despite the title on her book, “It’s funny, I don’t actually think of Weapons of Math Destruction as a book about math. It more about instruments of social control and how they are masquerading as an mathematical equation.” In her article, she talks about how there is a common belief among people that algorithms are unbiased because they are mathematical and don’t have agendas. However, she says the data itself cannot be decontextualized from our historical practices and the people that create them and put them in place often have prejudices like anybody else. “In other words, we don’t move past discrimination with the use of algorithms, but rather instead sanitize and obscure our historical-cultural practices and patterns.” She describes these “weapons of math destruction” as widespread, mysterious and destructive. She says widespread because she is only concerned about the algorithms that can affect a large number of people. They are also mysterious because different companies have a “secret sauce” that targets certain groups of people. O’Neil says that these groups of people are usually poor people and those of color. She explains there is a scoring system that targets them, they don’t understand how they are being scored and they are unaware that they are even being scored at all.
As an example of a “weapon of math destruction” O’Neil talks about the first one she came across outside of finance, called the teacher value-added model. She calls it an opaque system that uses student test scores in order to assess teacher ability. She says that this model is unreliable. However, it is still used in high stakes decisions such as if a teacher should be fired for her scores not being high enough. O’Neil interviewed a teacher that had been fired for having a score that was too low, that the teacher believed was caused by a previous teacher’s cheating. A version of the teacher value-added model is being used all around the country and they have a bias towards urban school districts. The model is causing the best teachers to leave the poorest neighborhoods, not the worst.