The Algorithm Audit: Scoring the algorithms that score us

by Shea Brown

Big Data & Society. doi:10.1177/2053951720983865. First published: January 28th 2021.

“The Algorithm Audit: Scoring the algorithms that score us” outlines a conceptual framework for auditing algorithms for potential ethical harms. In recent years, the ethical impact of AI has been increasingly scrutinized, and has led to a growing mistrust of AI and increased calls for mandated audits of algorithms. While there are many excellent proposals for ethical assessments of algorithms, such as Algorithmic Impact Assessments or the similar Automated Decision System Impact Assessments, these are too high level to be put directly into practice without further guidance. Other proposals have a more narrow focus on technical notions of bias or transparency (Mitchell et al., 2019). Moreover, without a unifying conceptual framework for carrying out these evaluations, there’s a worry that the ad hoc nature of the methodology could lead to potential harms being missed. 

We present an auditing framework that can serve as a more practical guide for comprehensive ethical assessments of algorithms. We clarify what we mean by an algorithm audit, explain key preliminary steps to any such audit (identifying the purpose of the audit, describing and circumscribing its context) and elaborate on the three main elements of the audit instrument itself: (i) a list of possible interests and rights of stakeholders affected by the algorithm, (ii) a list and assessment of metrics that describe key ethically salient features of the algorithm in the relevant context, and (ii) a relevancy matrix that connects the assessed metrics to the stakeholder interests.  We provide a simple example to illustrate how the audit is supposed to work, and discuss the different forms the audit result could take (quantitative score, qualitative score, and a narrative assessment).  

Our motivations for this separation of descriptive (metrics) and normative (interests) features are many, but one important reason is that this separation forces an auditor to carefully consider each stakeholder explicitly, and consider the possible relevance of various features of the algorithm (metrics) to that stakeholder’s interests. It’s important to note that different stakeholders in the same category (e.g. students, loan applicants, those up for parole, etc.) are often affected in very different ways by the same algorithm and often on the basis of race, ethnicity, gender, age, religion, or sexual orientation (Benjamin, 2019). We argue that understanding the context of an algorithm is a precursor to being able to not only enumerate stakeholder interests generally, but also to be able to identify particular sub-categories of stakeholders whose identification is relevant for ethical assessment of an algorithm (e.g. students of color, Hispanic loan applicants, male African-Americans up for parole, etc.). These stakeholders might face particular threats, and attention to context allows us to guard against thinking of groups of stakeholders are homogeneous entities that will be negatively or positively affected simply in virtue of the type of engagement with an algorithm, and to recognize socio-political and socio-technical factors, and power dynamics at play (Benjamin, 2019; D’Ignazio and Klein, 2020; Mohamed et al., 2020).

The proposed audit instrument yields an ethical evaluation of an algorithm that could be used by regulators and others interested in doing due diligence, while paying careful attention to the complex societal context within which the algorithm is deployed. It can also help institutions mitigate the reputational, financial, and ethical risk that a poorly performing algorithm might present.  

  

Revisiting the Black Box Society by Rethinking the Political Economy of Big Data

Special Theme Issue
https://journals.sagepub.com/page/bds/collections/revisitingtheblackboxsociety

Guest lead editors: Benedetta Brevini* and Frank Pasquale**

* University of Sydney
** Brooklyn Law School

Throughout the 2010s, scholars explored the politics and sociology of data, its regulation and its role in informing and guiding policymakers such as the importance of quality health data in the COVID-19 epidemic to “flatten the curve.” However, all too much of this work is being done in “black box societies” jurisdictions where the analysis and use of data is opaque, unverifiable, and unchallengeable. As a result, far too often data are used as a tool for social, political, and economic control, with biases often distorting decision making and accompanied by narratives of tech solutionism and even salvation-ism abound.

The Black Box Society was one of first scholarly accounts of algorithmic decision making to synthesize empirical research, normative frameworks, and legal argument and this symposium of commentaries reflect on what has happened since its publication. Much has happened since 2015 that vindicates and challenges the book’s main themes. Yet recurring examples of algorithmically driven injustices raise the question of whether transparency—the foundational normative value in The Black Box Society—is a first step toward a more emancipatory deployment of algorithms and AI, is an easily deflected demand, or actually worsens matters by rationalizing the algorithmic ordering of human affairs.

To address these issues, this symposium features the work of leading thinkers who have explored the interplay of politics, economics, and culture in domains ordered algorithmically by managers, bureaucrats, and technology workers. By bringing social scientists and legal experts into dialogue, we aim both to clarify the theoretical foundations of critical algorithm studies and to highlight the importance of engaged scholarship, which translates the insights of the academy into an emancipatory agenda for law and policy reform. While the contributions are diverse, a unifying theme animates them: each offers a sophisticated critique of the interplay between state and market forces in building or eroding the many layers of our common lives, as well as the kaleidoscopic privatization of spheres of reputation, search, and finance. Unsatisfied with narrow methodologies of economics or political science, they advance politico-economic analysis. They therefore succeed in unveiling the foundational role that the turn to big data has in organising economic and social relations. All the contributors help us imagine practical changes to prevailing structures that will advance social and economic justice, mutual understanding, and ecological sustainability. For this and much else, we are deeply grateful for their insightful work.

Editorial by Benedetta Brevini and Frank Pasquale, "Revisiting the Black Box Society by rethinking the political economy of big data". 

Ifeoma Ajunwa, in “The Black Box at Work,” describes the data revolution of the workplace, which simultaneously demands workers surrender intimate data and then prevents them from reviewing how it is used.

Mark Andrejevic, in “Shareable and Un-Shareable Knowledge,” focuses on what it means to generate actionable but non-shareable information, reaffirming the urgency of intelligible evaluation as a form of dignity.

Margaret Hu’s article “Cambridge Analytica’s Black Box” surveys a range of legal and policy remedies that have been proposed to better protect consumer data and informational privacy.

Paul Prinsloo examines “Black Boxes and Algorithmic Decision-making in (Higher) Education” to show how the education sector is beginning to adopt technologies of monitoring and personalization that are similar to the way the automated public sphere serves political information to voters.

Benedetta Brevini, in “Black Boxes, not Green: Mythologizing AI and Omitting the Environment” documents how AI runs on technology, machines and infrastructures that deplete scarce resources in their production, consumption and disposal, thus placing escalating demands on energy and accelerating the climate emergency.

Gavin Smith develops the concept of our “right to the face” in “The Face is the Message: Theorisingthe Politics of Algorithmic Governance in the Black Box City” as he explores how algorithms are now responsible for important surveillance of cities, constantly passing judgment on mundane activities.

Nicole Dewandre’s article, “Big Data: From Fears of the Modern to Wake-up Call for a New Beginning” applies a deeply nuanced critique of modernity to algorithmic societies arguing that Big Data may be hailed as the endpoint or materialisation of a Western modernity, or as a wake-up call for a new beginning.

Jonathan Obar confirms this problem empirically in “Sunlight Alone is Not a Disinfectant: Consent andthe Futility of Opening Big Data Black Boxes,” and proposes solutions to more equitably share the burden of understanding.

Kamel Ajji in “CyborgFinance Mirrors Cyborg Social Media” outlines how The Black Box Society inspired him to found “21 Mirrors, a nonprofit organization aimed at analyzing, rating and reporting to the public about the policies and practices of social media, web browsers and email services regarding their actual and potential consequences on freedom of expression, privacy, and due process.”

Designing for human rights in AI

Evgeni Aizenberg and Jeroen van den Hoven introduce their publication "Designing for human rights in AI" in Big Data & Society 7(2), https://doi.org/10.1177/2053951720949566. First published: Aug 18, 2020.

Video abstract

Text abstract
In the age of Big Data, companies and governments are increasingly using algorithms to inform hiring decisions, employee management, policing, credit scoring, insurance pricing, and many more aspects of our lives. Artificial intelligence (AI) systems can help us make evidence-driven, efficient decisions, but can also confront us with unjustified, discriminatory decisions wrongly assumed to be accurate because they are made automatically and quantitatively. It is becoming evident that these technological developments are consequential to people’s fundamental human rights. Despite increasing attention to these urgent challenges in recent years, technical solutions to these complex socio-ethical problems are often developed without empirical study of societal context and the critical input of societal stakeholders who are impacted by the technology. On the other hand, calls for more ethically and socially aware AI often fail to provide answers for how to proceed beyond stressing the importance of transparency, explainability, and fairness. Bridging these socio-technical gaps and the deep divide between abstract value language and design requirements is essential to facilitate nuanced, context-dependent design choices that will support moral and social values. In this paper, we bridge this divide through the framework of Design for Values, drawing on methodologies of Value Sensitive Design and Participatory Design to present a roadmap for proactively engaging societal stakeholders to translate fundamental human rights into context-dependent design requirements through a structured, inclusive, and transparent process.

Keywords: Artificial intelligence, human rights, Design for Values, Value Sensitive Design, ethics, stakeholders

Establishing a Social Licence for FinTech: Reflections on the role of the private sector in pursuing ethical data practices

Mhairi Aitken, Ehsan Toreini, Peter Carmichael, Kovila Coopamootoo, Karen Elliott, Aad van Moorsel
Big Data & Society 7(1), https://doi.org/10.1177/2053951720908892. First published: March 4, 2020
Keywords: Financial Technology, data, social licence, ethics, responsible artificial intelligence, trust

Recent years have witnessed a dramatic increase in attention directed at ethical dimensions of data practices and Artificial Intelligence (AI). Increasingly momentum for innovation is being met with interest in related ethical considerations and a number of high profile institutes and bodies have been established to focus on this area. The substantial investment in this field has to date largely resulted in a proliferation of guidance and sets of principles relating to ethical AI but important questions remain as to how such principles can be put into practice, and to what extent commitments to ethical AI go beyond rhetoric.

These are questions we engage with in our paper “Establishing a Social Licence for FinTech” and which also underpin our ongoing programme of research through our EPSRC-funded project “FinTrust” which examines the role of AI in finance.

We focus on FinTech (financial technology) as this represents a fast-moving industry and one which is attracting substantial investment. Within FinTech there is industrial advocacy surrounding the potential benefits of data science and AI in banking, however to date there has been little consideration of the ethical dimensions of these practices or the extent to which they align with public values and expectations. Therefore, our research focusses on FinTech in order to examine the opportunities and potential approaches to develop ethical data practices which go beyond compliance with regulation.

In our paper we consider the importance of establishing and maintaining a Social Licence for data practices. The notion of a Social Licence recognises that there can be meaningful differences between what is legally permissible and what is socially acceptable. A Social Licence is granted by a community of stakeholders and is intangible and unwritten but may be essential for the sustainability and legitimacy of particular practices or industries.

With attention being directed at digital ethics there is emerging interest in pursuing a Social Licence for data practices. However, it is interesting that while the notion of a Social Licence emerged in the 1990s in relation to private sector extractive industries (e.g. mining and forestry), to date where this has been discussed with regards data practices it has largely been in relation to public sector activities (e.g. healthcare and health research). In our paper we therefore consider what this means for private sector data-intensive industries, such as FinTech.

In discussing what would be required to establish a Social Licence for FinTech, we consider three main points:

1. A Social Licence is underpinned by relationships of trust which need to be sustained over time. We consider how trust is established and what it might mean for a FinTech to be considered trustworthy.
2. Establishing trust requires both technical and social approaches. We discuss the current technical approaches advocated in ethical AI (relating to Robustness, Fairness, Explainability and Lineage), the extent to which they may be conceived to demonstrate trustworthiness, and the importance of combining these with social approaches.
3. Establishing and maintaining a Social Licence requires engagement with diverse stakeholders. Given that data practices are having far-reaching – and often unpredicted – impacts across society a broad conception of stakeholders acknowledges the importance of wide public engagement beyond potential service-users. We suggest that wide public engagement with broad publics is vital to ensure that current and future practices reflect public values and interests. Our paper then considers the extent to which it is reasonable to expect such broad approaches to be adopted by individual FinTechs or the wider industry.

The paper poses a number of questions to which we do not yet have the answers. For example, the paper does not aim to identify public interests or concerns relating to data practices in FinTech, or to set out what is required for FinTech to align with public values. Since there is a paucity of public engagement or deliberation examining public values around FinTech practices, further research (including through public engagement methods) is needed to examine what this means in practice.

Combining our interdisciplinary perspectives from Computer Science, Sociology, Human Computer Interaction and Organisational Science, our FinTrust project is continuing to build on the work presented in this paper to address these tricky questions. We aim to develop a toolkit which will set out a combination of technical and social approaches to underpin a future Social Licence for FinTech practices.

We posit that such approaches are needed across all areas and industries whose operations are dependent on data. Pursuing a Social Licence will complement regulation and build on ethical codes of practice. This is important to underpin culture change and to move beyond rhetorical commitments to develop best practice, meaningfully putting ethics at the heart of innovation.