Den mörka sidan av AI inom public service

Artificial intelligence and machine learning have the potential to revolutionize the way we live and work, but as we increasingly rely on these technologies, it’s crucial that we ensure they are designed, implemented, and monitored in an ethical manner. Unfortunately, this is not always the case, as numerous examples of the dark side of AI from around the world show. From the biased AI system used to predict crime in Chicago, to the AI-powered welfare assessments in Australia, and the use of AI in the criminal justice system in Germany, these technologies have too often perpetuated existing biases and inequalities. In this article, we will examine some of these examples and discuss what governments and societies need to do in order to prevent future AI-powered scandals and reduce societal and governmental blind spots.

What are societal blind spots?

Blind spots refer to areas of society where individuals or groups hold biases or lack sufficient information or understanding. In politics and society, these blind spots can manifest in a number of ways, including:

1. Lack of representation: Blind spots in politics can occur when certain groups, such as women, people of color, or the LGBTQ+ community, are underrepresented in political decision-making and do not have their perspectives and experiences taken into account.
2. Unconscious bias: Societal blind spots can also result from unconscious biases, where individuals hold preconceived notions about certain groups that shape their perceptions and actions, even if they are not aware of it.
3. Limited understanding: Blind spots can also result from a limited understanding of certain social issues or experiences, such as poverty or disabilities. This can lead to inadequate policies or solutions that fail to address the root causes of these issues.
4. Echo chambers: In the age of social media, individuals are often exposed to information and perspectives that align with their own beliefs and values, creating “echo chambers” where it is difficult to challenge one’s own biases and assumptions.

These blind spots in politics and society can have negative consequences, including perpetuating inequality and injustice, and hindering progress towards creating a more inclusive and just society. Addressing these blind spots requires continuous effort to increase representation and understanding, challenge biases and assumptions, and seek out diverse perspectives and experiences.

The Dangers of Unregulated AI Adoption: Perpetuating Bias and Amplifying Blind Spots

The massive adoption of AI and machine learning, as well as increasing automation by governments, could potentially lead to bigger blind spots in society. This is because AI systems are only as good as the data and algorithms that drive them, and if these systems are not designed and implemented with diversity, fairness, and transparency in mind, they can perpetuate or amplify existing biases and injustices.

For example, if an AI system is trained on biased data, it will make decisions that reflect those biases, potentially leading to unequal outcomes. Furthermore, if the decision-making process of AI systems is not transparent, it can be difficult to identify and address sources of bias, which can further exacerbate existing blind spots.

Examples of Government AI gone Awry

Netherlands

The child benefit (dutch: toeslagen affaire) scandal in the Netherlands is an example of how the use of AI and machine learning in government decision-making can lead to societal blind spots and serious consequences.

In this case, the Dutch government used an AI system to detect fraud in the child benefit (toeslagenschandaal) program. However, the system was based on incorrect assumptions and was not properly tested, leading to thousands of families being wrongly accused of fraud and having their benefits taken away. The scandal resulted in widespread public outrage, as many of the affected families were struggling to make ends meet and were not able to easily contest the decision.

The child benefit scandal highlights the importance of considering societal blind spots when implementing AI and machine learning systems in government decision-making. In this case, the government did not properly account for the experiences and perspectives of low-income families, leading to a system that was not only inaccurate but also had devastating consequences for those affected.

United States

In the United States, the city of Chicago implemented an AI system to predict where crime would occur, but the system was found to have a significant bias against African American communities. This resulted in increased police surveillance in those areas, which caused harm to the communities and contributed to a vicious cycle of over-policing and over-criminalization. The bias in the AI system was due to the use of historical crime data, which was skewed to reflect over-policing in African American communities.

India

In India, AI was introduced to evaluate the financial health of small businesses and determine loan approvals, but it was discovered that the system was biased against businesses owned by women and low-caste individuals. This resulted in unequal access to financing and perpetuated existing societal inequalities. The bias in the AI system was due to a lack of representation of these groups in the training data, leading to an inaccurate assessment of their financial health. 

United Kingdom

In the United Kingdom, an AI system was introduced to determine eligibility for disability benefits, but it was soon discovered that the system was inaccurate and insensitive. This led to many people with disabilities being wrongly denied benefits and suffering financial hardship. The inaccuracies in the AI system were due to a lack of understanding of the complex needs of individuals with disabilities and a reliance on flawed data and processes.

Australia

In Australia, an AI system was introduced to determine the eligibility of welfare recipients, but it was soon discovered to be overly punitive. This resulted in many people being wrongly denied benefits or having their benefits reduced, causing significant financial hardship and stress. The harsh approach of the AI system was due to a lack of understanding of the complexities of people’s lives and an over-reliance on strict rules and algorithms.

Germany

In Germany, an artificial intelligence system that was designed to evaluate the risk of recidivism among criminal defendants has been discovered to have a bias against individuals of color. This has resulted in unequal treatment within the country’s justice system, where people of color are more likely to be deemed high-risk and face harsher sentencing or denial of parole compared to their white counterparts with similar records.

Canada

In Canada, an artificial intelligence system that was utilized to determine immigration status and eligibility for benefits was found to have widespread inaccuracies. This resulted in numerous individuals being wrongly deported or denied benefits, leading to devastating consequences for their lives and families.

Fair AI for a Better Future

Artificial Intelligence (AI) has the potential to revolutionize the way governments and societies operate, but it is crucial that steps are taken to ensure its use is fair, accurate, and unbiased. To achieve this, several key factors must be considered and addressed.

1. Increased transparency and accountability in the development and deployment of AI systems.
2. Ongoing monitoring and evaluation to ensure that AI systems are working fairly and accurately.
3. Representation of diverse perspectives and experiences in the development and deployment of AI systems.
4. Inclusion of individuals from different races, genders, and socio-economic backgrounds in the development process.
5. Addressing potential biases and errors in AI systems, especially in sensitive areas such as criminal justice, welfare, and employment.

Essential reading

For readers interested in further exploring the impact of the dark side of AI, the book ‘Weapons of Math Destruction’ by Cathy O’Neil is a valuable resource. It highlights the ways in which mathematical models can perpetuate and amplify social inequalities, and provides recommendations for how to create more responsible and equitable use of data and technology.