The Algorithm Audit: Scoring the algorithms that score us

by Shea Brown

Big Data & Society. doi:10.1177/2053951720983865. First published: January 28th 2021.

“The Algorithm Audit: Scoring the algorithms that score us” outlines a conceptual framework for auditing algorithms for potential ethical harms. In recent years, the ethical impact of AI has been increasingly scrutinized, and has led to a growing mistrust of AI and increased calls for mandated audits of algorithms. While there are many excellent proposals for ethical assessments of algorithms, such as Algorithmic Impact Assessments or the similar Automated Decision System Impact Assessments, these are too high level to be put directly into practice without further guidance. Other proposals have a more narrow focus on technical notions of bias or transparency (Mitchell et al., 2019). Moreover, without a unifying conceptual framework for carrying out these evaluations, there’s a worry that the ad hoc nature of the methodology could lead to potential harms being missed. 

We present an auditing framework that can serve as a more practical guide for comprehensive ethical assessments of algorithms. We clarify what we mean by an algorithm audit, explain key preliminary steps to any such audit (identifying the purpose of the audit, describing and circumscribing its context) and elaborate on the three main elements of the audit instrument itself: (i) a list of possible interests and rights of stakeholders affected by the algorithm, (ii) a list and assessment of metrics that describe key ethically salient features of the algorithm in the relevant context, and (ii) a relevancy matrix that connects the assessed metrics to the stakeholder interests.  We provide a simple example to illustrate how the audit is supposed to work, and discuss the different forms the audit result could take (quantitative score, qualitative score, and a narrative assessment).  

Our motivations for this separation of descriptive (metrics) and normative (interests) features are many, but one important reason is that this separation forces an auditor to carefully consider each stakeholder explicitly, and consider the possible relevance of various features of the algorithm (metrics) to that stakeholder’s interests. It’s important to note that different stakeholders in the same category (e.g. students, loan applicants, those up for parole, etc.) are often affected in very different ways by the same algorithm and often on the basis of race, ethnicity, gender, age, religion, or sexual orientation (Benjamin, 2019). We argue that understanding the context of an algorithm is a precursor to being able to not only enumerate stakeholder interests generally, but also to be able to identify particular sub-categories of stakeholders whose identification is relevant for ethical assessment of an algorithm (e.g. students of color, Hispanic loan applicants, male African-Americans up for parole, etc.). These stakeholders might face particular threats, and attention to context allows us to guard against thinking of groups of stakeholders are homogeneous entities that will be negatively or positively affected simply in virtue of the type of engagement with an algorithm, and to recognize socio-political and socio-technical factors, and power dynamics at play (Benjamin, 2019; D’Ignazio and Klein, 2020; Mohamed et al., 2020).

The proposed audit instrument yields an ethical evaluation of an algorithm that could be used by regulators and others interested in doing due diligence, while paying careful attention to the complex societal context within which the algorithm is deployed. It can also help institutions mitigate the reputational, financial, and ethical risk that a poorly performing algorithm might present.  

  

COVID-19 is spatial: Ensuring that mobile Big Data is used for social good

by Age Poom, Olle Järv, Matthew Zook and Tuuli Toivonen

Big Data & Society 7(2), https://doi.org/10.1177/2053951720952088. First published: August 28, 2020

The mobility restrictions related to COVID-19 pandemic have resulted in the biggest disruption to individual mobilities in modern times. Hot spots, quarantine, closed borders, video-conferencing, social distancing and temporary closure of workplaces, schools, restaurants and recreational facilities are all profoundly about distance, separation, and space. Examining the geographical aspect of the pandemic is important in understanding its broad implications, including the broader societal impacts of containment policies. 

The avalanche of mobile Big Data – location and time-stamp data from mobile phone call records, operating system, social media or apps – makes it possible to study the spatial effects of the crisis with spatiotemporal detail even at national and global scales. Beyond health care objectives such as understanding how virus transmission is mediated by human mobility or evaluating adherence to restrictions, mobile Big Data also allows us to understand the changes in people’s daily interactions, mobilities and socio-spatial responses across population groups.

Our advocacy for the use of these data, however, is tempered both by our experiences in recent months with the serious limitations of using mobile Big Data and our unease with the power of these same data to track, surveil and discipline social behaviour at the scale of entire populations. 

Thus, we pose the question: How can we use mobile Big Data for social good, while also protecting society from social harm? Drawing on the Estonian and Finnish experiences during the early phases of COVID-19 pandemic, we highlight issues with quickly developed ad hoc data products as well as the “black box” solutions (Pasquale, 2015) offered by large platform companies that created “new digital divides” among researchers (boyd and Crawford, 2012).

We argue that these examples demonstrate a clear need to re-evaluate the public-private relationships with mobile Big Data and propose two strategic pathways forward.

First, we call for transparent and sound mobile Big Data products that provide relevant up-to-date longitudinal data on the mobility patterns of dynamic populations. To help increase their usefulness, data products should be transparent about their production methodology, and ensure easy access and stability. 

Second, there is also a need to develop trustworthy platforms for collaborative use of raw individual level data. Secured and privacy-respectful access to near real-time raw data is needed for developing and testing sound methodologies for the above-mentioned data products. This would help bridge the Big Data digital divide, enable scientific innovation, and offering needed flexibility in responding to unanticipated questions on changing locations and mobilities in case of crises. To be clear, we do not view this as simple to achieve, particularly as we weigh what kind of institution might best fill this role, or how is “social good” defined and operationalized in practice. But addressing these issues via public debates and academic discourses will leave us better prepared for the next crisis.

Summing up,

• We need harmonized and representative data about human mobility for better crisis preparedness and social good in general;
• Methodological transparency about mobile Big Data products are vital for open societies and capacity building;
• Access to mobile Big Data to develop feasible methodologies and baseline knowledge for public decision-making is needed before the next crisis occurs;
• Recognizing the fundamental spatiality of the current COVID-19 crisis and crises more generally is the most relevant of all.

Mobile Big Data can help us to better understand and address the important spatial dimensions of COVID-19 pandemic and every other social phenomenon. The challenge is doing so responsibly (Zook et al., 2017) and not normalizing a lack of spatial privacy.

References

boyd, d, Crawford, K (2012) Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon. Information, Communication & Society 15(5): 662–679. https://doi.org/10.1080/1369118X.2012.678878

Pasquale, F (2015) The Black Box Society. Cambridge: Harvard University Press.

Zook, M, Barocas, S, boyd, d, et al. (2017) Ten simple rules for responsible big data research. PLOS Computational Biology 13(3): e1005399. https://doi.org/10.1371/journal.pcbi.1005399

Keywords: mobile Big Data, mobility, COVID-19, spatial data infrastructure, social good, mobile phone data, social media data, privacy

Designing for human rights in AI

Evgeni Aizenberg and Jeroen van den Hoven introduce their publication "Designing for human rights in AI" in Big Data & Society 7(2), https://doi.org/10.1177/2053951720949566. First published: Aug 18, 2020.

Video abstract

Text abstract
In the age of Big Data, companies and governments are increasingly using algorithms to inform hiring decisions, employee management, policing, credit scoring, insurance pricing, and many more aspects of our lives. Artificial intelligence (AI) systems can help us make evidence-driven, efficient decisions, but can also confront us with unjustified, discriminatory decisions wrongly assumed to be accurate because they are made automatically and quantitatively. It is becoming evident that these technological developments are consequential to people’s fundamental human rights. Despite increasing attention to these urgent challenges in recent years, technical solutions to these complex socio-ethical problems are often developed without empirical study of societal context and the critical input of societal stakeholders who are impacted by the technology. On the other hand, calls for more ethically and socially aware AI often fail to provide answers for how to proceed beyond stressing the importance of transparency, explainability, and fairness. Bridging these socio-technical gaps and the deep divide between abstract value language and design requirements is essential to facilitate nuanced, context-dependent design choices that will support moral and social values. In this paper, we bridge this divide through the framework of Design for Values, drawing on methodologies of Value Sensitive Design and Participatory Design to present a roadmap for proactively engaging societal stakeholders to translate fundamental human rights into context-dependent design requirements through a structured, inclusive, and transparent process.

Keywords: Artificial intelligence, human rights, Design for Values, Value Sensitive Design, ethics, stakeholders

Establishing a Social Licence for FinTech: Reflections on the role of the private sector in pursuing ethical data practices

Mhairi Aitken, Ehsan Toreini, Peter Carmichael, Kovila Coopamootoo, Karen Elliott, Aad van Moorsel
Big Data & Society 7(1), https://doi.org/10.1177/2053951720908892. First published: March 4, 2020
Keywords: Financial Technology, data, social licence, ethics, responsible artificial intelligence, trust

Recent years have witnessed a dramatic increase in attention directed at ethical dimensions of data practices and Artificial Intelligence (AI). Increasingly momentum for innovation is being met with interest in related ethical considerations and a number of high profile institutes and bodies have been established to focus on this area. The substantial investment in this field has to date largely resulted in a proliferation of guidance and sets of principles relating to ethical AI but important questions remain as to how such principles can be put into practice, and to what extent commitments to ethical AI go beyond rhetoric.

These are questions we engage with in our paper “Establishing a Social Licence for FinTech” and which also underpin our ongoing programme of research through our EPSRC-funded project “FinTrust” which examines the role of AI in finance.

We focus on FinTech (financial technology) as this represents a fast-moving industry and one which is attracting substantial investment. Within FinTech there is industrial advocacy surrounding the potential benefits of data science and AI in banking, however to date there has been little consideration of the ethical dimensions of these practices or the extent to which they align with public values and expectations. Therefore, our research focusses on FinTech in order to examine the opportunities and potential approaches to develop ethical data practices which go beyond compliance with regulation.

In our paper we consider the importance of establishing and maintaining a Social Licence for data practices. The notion of a Social Licence recognises that there can be meaningful differences between what is legally permissible and what is socially acceptable. A Social Licence is granted by a community of stakeholders and is intangible and unwritten but may be essential for the sustainability and legitimacy of particular practices or industries.

With attention being directed at digital ethics there is emerging interest in pursuing a Social Licence for data practices. However, it is interesting that while the notion of a Social Licence emerged in the 1990s in relation to private sector extractive industries (e.g. mining and forestry), to date where this has been discussed with regards data practices it has largely been in relation to public sector activities (e.g. healthcare and health research). In our paper we therefore consider what this means for private sector data-intensive industries, such as FinTech.

In discussing what would be required to establish a Social Licence for FinTech, we consider three main points:

1. A Social Licence is underpinned by relationships of trust which need to be sustained over time. We consider how trust is established and what it might mean for a FinTech to be considered trustworthy.
2. Establishing trust requires both technical and social approaches. We discuss the current technical approaches advocated in ethical AI (relating to Robustness, Fairness, Explainability and Lineage), the extent to which they may be conceived to demonstrate trustworthiness, and the importance of combining these with social approaches.
3. Establishing and maintaining a Social Licence requires engagement with diverse stakeholders. Given that data practices are having far-reaching – and often unpredicted – impacts across society a broad conception of stakeholders acknowledges the importance of wide public engagement beyond potential service-users. We suggest that wide public engagement with broad publics is vital to ensure that current and future practices reflect public values and interests. Our paper then considers the extent to which it is reasonable to expect such broad approaches to be adopted by individual FinTechs or the wider industry.

The paper poses a number of questions to which we do not yet have the answers. For example, the paper does not aim to identify public interests or concerns relating to data practices in FinTech, or to set out what is required for FinTech to align with public values. Since there is a paucity of public engagement or deliberation examining public values around FinTech practices, further research (including through public engagement methods) is needed to examine what this means in practice.

Combining our interdisciplinary perspectives from Computer Science, Sociology, Human Computer Interaction and Organisational Science, our FinTrust project is continuing to build on the work presented in this paper to address these tricky questions. We aim to develop a toolkit which will set out a combination of technical and social approaches to underpin a future Social Licence for FinTech practices.

We posit that such approaches are needed across all areas and industries whose operations are dependent on data. Pursuing a Social Licence will complement regulation and build on ethical codes of practice. This is important to underpin culture change and to move beyond rhetorical commitments to develop best practice, meaningfully putting ethics at the heart of innovation.

Engaging with ethics in Internet of Things: Imaginaries in the social milieu of technology developers

Funda Ustek-Spilda, Alison Powell, Selena Nemorin
Big Data & Society 6(2), https://doi.org/10.1177/2053951719879468. First published Oct 3, 2019.
Keywords: Internet of Things, social milieu, ethics, virtue ethics, responsible technology

Discussions about ethics of Big Data often focus on the ethics of data processing – ‘generation, recording, curation, processing, dissemination, sharing and use’ of algorithms (including machine learning and artificial intelligence) as well as corresponding practices such as programming, hacking and coding (Floridi and Taddeo, 2016: 1). Data-based systems, however, do not come from nowhere. In this article, we attempt to shift the focus of ethical discussion from the context of data processing to the contexts of data production. We attend to the ethical qualities of the social milieu in which data-intensive technologies get to be produced and the practical reasoning people in this social milieu undertake in their day-to-day encounters with technology development.

Our analysis is based on our ongoing work as part of a research project titled VIRT-EU: Values and Ethics for Responsible Innovation in EUrope. As part of our research, we have conducted multi-site ethnographic fieldwork with developers, designers and entrepreneurs as part of the IoT startup ecosystem in Europe. Between 2016 and 2018, we followed industry meetups, hardware and software showcases, workshops and industry conferences and conducted in person interviews and held co-design workshops; amounting to more than 100 unique fieldwork visits. We also analysed 10 years of data from the records of the IoT meetups in Europe. In conducting our analysis, we sought answers to the following two questions: How do developers in start-ups and small companies practice ethical decision-making? What are the technological, business and social contexts that influence these decisions?

Our findings indicated that the social milieu of technology development, being strongly focused on innovation, attracting funding, corporate reputation and market share created challenges for explicit engagement with ethics. This, we argued, holds a major constraint to systemic change in the field. Many people considered ethics as important as a topic, but not urgent in their list of things to-do. From our analysis, we developed three action positions to illustrate points of engagement with ethical and moral concerns. These positions are of course not exhaustive of the positions available to those in these spaces, but include the most significant directions of engagements we observed in our fieldwork. These positions are the Disengaged; the Pragmatist and the Idealist. Within the Disengaged position, many IoT developers remained ambivalent about the 'use' of ethical reflection and discussion beyond compliance with existing regulations; concentrating their attention more on issues relating to business and financial stability. To illustrate, within the nearly 90 meetups held by IoT London Meetup Group held in the last ten years, our analysis indicated that ethics as a topic featured only once, while GDPR emerged as a topic that was mentioned often. The Pragmatist position places ethical concerns squarely in relation to business interests but is not necessarily subsumed by them. We found that ethics was referred to in its relation to new and emerging market opportunities and allowing businesses to limit financial liability. An Idealist position on the other hand, advocated action on values and principles by incorporating them directly into business ventures and social networks. A series of IoT manifestos advanced some of these perspectives (Fritsch et al., 2018) and some developers we interviewed also positioned themselves and the trajectories of their ventures along these lines. A strong identification with ‘we’ rather than ‘I’ and separation of individual and collective subjectivities in relation to ethical concerns as well as an active engagement with the responsibility for producing ethical technologies (and futures) were shared among these individuals.

Our analysis demonstrates that the extent to which individual subjectivity can influence engaging in ethical action may depend on the organisational environment technology developers are embedded in. This means that constraints (financial, structural, social or other) are not merely external things to be overcome for ethical action to take place, but rather intrinsic to the social milieu technology developers are part of. This goes some way to explain why on the one hand we are seeing a plethora of new ventures subscribing to emerging fields such as ‘technology for social good’ or ‘business with purpose’ whilst on the other hand technology products continue to violate privacy, intensify bias and entrench social power. Put simply, it is not simply that technology developers do not have ‘virtuous intentions’ but that the social milieu they are part of structures their space for action.

References

Floridi L and Taddeo M (2016) What is data ethics? Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 374(2083): 20160360.

Fritsch E, Shklovski I and Douglas-Jones R (2018) Calling for a revolution: An analysis of IoT manifestos. In: Proceedings of the 2018 CHI conference on human factors in computing systems, Montreal, QC, Canada, 21–26 April 2018, p.302. New York: ACM Press.