The Algorithm Audit: Scoring the algorithms that score us

by Shea Brown

Big Data & Society. doi:10.1177/2053951720983865. First published: January 28th 2021.

“The Algorithm Audit: Scoring the algorithms that score us” outlines a conceptual framework for auditing algorithms for potential ethical harms. In recent years, the ethical impact of AI has been increasingly scrutinized, and has led to a growing mistrust of AI and increased calls for mandated audits of algorithms. While there are many excellent proposals for ethical assessments of algorithms, such as Algorithmic Impact Assessments or the similar Automated Decision System Impact Assessments, these are too high level to be put directly into practice without further guidance. Other proposals have a more narrow focus on technical notions of bias or transparency (Mitchell et al., 2019). Moreover, without a unifying conceptual framework for carrying out these evaluations, there’s a worry that the ad hoc nature of the methodology could lead to potential harms being missed. 

We present an auditing framework that can serve as a more practical guide for comprehensive ethical assessments of algorithms. We clarify what we mean by an algorithm audit, explain key preliminary steps to any such audit (identifying the purpose of the audit, describing and circumscribing its context) and elaborate on the three main elements of the audit instrument itself: (i) a list of possible interests and rights of stakeholders affected by the algorithm, (ii) a list and assessment of metrics that describe key ethically salient features of the algorithm in the relevant context, and (ii) a relevancy matrix that connects the assessed metrics to the stakeholder interests.  We provide a simple example to illustrate how the audit is supposed to work, and discuss the different forms the audit result could take (quantitative score, qualitative score, and a narrative assessment).  

Our motivations for this separation of descriptive (metrics) and normative (interests) features are many, but one important reason is that this separation forces an auditor to carefully consider each stakeholder explicitly, and consider the possible relevance of various features of the algorithm (metrics) to that stakeholder’s interests. It’s important to note that different stakeholders in the same category (e.g. students, loan applicants, those up for parole, etc.) are often affected in very different ways by the same algorithm and often on the basis of race, ethnicity, gender, age, religion, or sexual orientation (Benjamin, 2019). We argue that understanding the context of an algorithm is a precursor to being able to not only enumerate stakeholder interests generally, but also to be able to identify particular sub-categories of stakeholders whose identification is relevant for ethical assessment of an algorithm (e.g. students of color, Hispanic loan applicants, male African-Americans up for parole, etc.). These stakeholders might face particular threats, and attention to context allows us to guard against thinking of groups of stakeholders are homogeneous entities that will be negatively or positively affected simply in virtue of the type of engagement with an algorithm, and to recognize socio-political and socio-technical factors, and power dynamics at play (Benjamin, 2019; D’Ignazio and Klein, 2020; Mohamed et al., 2020).

The proposed audit instrument yields an ethical evaluation of an algorithm that could be used by regulators and others interested in doing due diligence, while paying careful attention to the complex societal context within which the algorithm is deployed. It can also help institutions mitigate the reputational, financial, and ethical risk that a poorly performing algorithm might present.  

  

How the pandemic made visible a new form of power

By Engin Isin and Evelyn Ruppert

The political pandemic known by its biomedical name COVID-19 has thrown us off balance. We have steadied ourselves (somewhat) by taking a historical view on forms of power and re-evaluating some long-held assumptions about power in social and political theory to see what the present might reveal. The result is this article where we broadly provide an account of three forms of power (sovereign, disciplinary and regulatory) and suggest that the coronavirus pandemic has brought these three forms of power into sharp relief while making particularly visible a fourth form of power that we name ‘sensory power’. We place sensory power within an historical series straddling the 17th and 18th (sovereign), 18th and 19th (disciplinary), and 19th and 20th (regulatory) centuries. The birth of sensory power straddles the 20th and 21st centuries and just like earlier forms of power neither replaces nor displaces but nestles and intertwines with them. So, rather than announcing the birth of a new form of power that marks this age or epoch we articulate its formation as part of a much more complex configuration of power. We conclude the article with a discussion the kinds of resistance that sensory power elicits.

This article follows the second edition of our book, Being Digital Citizens, which was published a few months earlier. Its new chapter examines various forms of digital activism. We were inspired by the range, spread, and diffusion of such acts of resistance and yet were unclear what form of power to which they were a response. Having examined these acts of resistance compelled us to take a historical view and to name a fourth form of power. With the onset of the pandemic, that form of power became visible and through the article we came to name it as sensory power.  

 

Both the article and the book are available at the links below and are open access.

 

Isin, Engin, and Evelyn Ruppert. 2020. ‘The Birth of Sensory Power: How a Pandemic Made It Visible’. Big Data & Society 7(2). Open access.

 

Isin, Engin F., and Evelyn Ruppert. 2020. Being Digital Citizens. Second edition. London: Rowman & Littlefield.  Go to tab 'features' to download open access pdf.

Automated Facts, Data Contextualization and Knowledge Colonialism: A Conversation Between Denny Vrandečić and Heather Ford on Wikipedia’s 20th Anniversary

At the start of 2021, contributors across the world will celebrate the 20-year anniversary of the first Wikipedia, and its 250,000 volunteers who supports it over 300 languages. In advance of Wikipedia’s 20th anniversary, Joseph Reagle, co-editor of Wikipedia @ 20: Stories of an Incomplete Revolution, 2020 MIT Press, facilitated a conversation between two of the volume’s essayists on the potential power and pitfalls of Wikipedia’s latest project: Abstract Wikipedia, an effort to automate the sharing of “facts”, information and data across different languages versions of the online encyclopedia.

Denny Vrandečić (founder of Wikidata and now Abstract Wikipedia, see FN1) and Heather Ford (scholar of Wikipedia governance) discuss the automation of digitised facts (e.g., Marie Curie’s birth year is 1867) as a means of reducing user effort and possible mistakes. Vrandečić and Ford discuss the two related concerns of when abstract “facts” are divorced from their context via automation and how the automated import of decontextualized data from larger projects can overwhelm smaller ones. Both of these concerns relate to the question of how automation influences peoples’ agency over the information that they produce at a local level. Automation may enhance efficiencies but this big data practice has important consequences for power and accountability within the world’s most widely used encyclopedia. 

------ Context ------

Ford: One of the key dangers with the automated extraction of facts from Wikipedia is that it has resulted in the loss of citation data. On Wikipedia, there is a significant focus on citing statements to reliable sources and this work is seen as central to the principle of verifiability. But on Wikidata, many statements are only cited to a particular Wikipedia language version because of their mass import. This ultimately prevents readers being able to follow up the source of a statement and influences the accountability of Wikimedia. How will Abstract Wikipedia avoid this problem?

Vrandečić: The problem of unsourced statements is not new to Wikidata. Wikidata makes it much more visible, though. As of 2020, more than 65% of  statements on Wikidata are sourced – this has gone up from less than 20% five years ago. Wikidata is improving considerably. I am actually convinced that the ratio of referenced statements on Wikidata is by now considerably better than on Wikipedia. But unfortunately, we are often compared with a perfect 100%, but I don’t think that is fair.

On Wikipedia it is much harder to count how many of its statements are actually sourced. But if you take a look at an average article on Wikipedia, you will find that far less than half of the statements in it have references – but we have no automatic metric counting this, so the actual comparison is rather hidden. I would love to see more research in this area.

Abstract Wikipedia will be more similar to Wikidata: it will be far easier to count how many statements have references than on Wikipedia, which will be the same blessing and curse as for Wikidata.

The other advantage is that due to the way Abstract Wikipedia is built, we will be able to generate sentences for each statement, and not just structured data as in Wikidata. This may in fact allow us to easier look for references in digital source material. Together with the fact that we can more easily see which statements are missing references, it should be possible to more easily create workflows that will search for possible references explicitly, and allow the community to fill the gaps.

Ford: You are right that the problem of unsourced statements is not new to Wikidata and it is true that it is much easier to figure out how much of Wikidata is unsourced than Wikipedia. But I don’t think that this is a reason to disavow the problem. The problem is significant: it influences Wikipedia’s core principles and affects how Wikipedia’s facts are ultimately understood – not as provisional statements that emerge from particular perspectives, but as hard facts that cannot be questioned. One could also argue that the problem is even more important to solve on Wikidata because of the authority that datafied facts have when they are extracted and presented on third party platforms.

I am also not as hopeful as you are that the problem is being solved over time. Having discussed this with others, I believe that the reason we are seeing improvement in the statistics is that WikiCite’s focus is on creating and maintaining source and citation data rather than annotating current Wikidata items. If you look at Wikidata items about places, Johannesburg, for example, you’ll notice that the majority of statements are not sourced meaningfully beyond a particular Wikipedia language version. A vast amount of manual labour is needed to do this work, and I just don’t see it happening anytime soon without a clear incentive and programme to do so at scale. I agree with you that we need more research in this area to look at the extent to which particular topics (such as places) are well-cited and how this has changed over time, but first it needs to be acknowledged that there is a potential problem that isn’t being dealt with.

I hope that Abstract Wikipedia prioritises and necessitates the addition of sources as new statements are added. If this can happen at the outset, with the small scale approach at the start, and if Abstract Wikipedia prioritises human review of sources, then it will certainly be an improvement on Wikidata. 

Vrandečić: I also hope that the Abstract Wikipedia community will put the right emphasis on sourcing its content, but in the end this is a decision to be made by that community, and not by me.

I think what helped with improving the situation on Wikidata so much over the last few years was that once the concern was raised, dashboards and metrics were introduced which made the situation more visible. I think the fact that the coverage of statements with sources more than tripled since then is a testament to the fact that this problem has been recognized and has been worked on by the community, and that the community is in fact very capable to deal with such issues.

Because of that I plan to put the same trust in the future community that will work on Abstract Wikipedia. And I welcome researchers like you to critically observe and accompany that development, after all it was exactly these concerns that lead to the set up of such dashboards and making these numbers more observable.

------ Colonialism ------

Ford: Abstract Wikipedia is intended to help populate smaller Wikipedias. How will you prevent the English-speaking community and other dominant languages from drowning out local content? For example, the encyclopedia in the Cebuano language, spoken in the southern Philippines, is one of the largest because it has been flooded by thousands of automatically translated articles, which is difficult for the local Cebuano community to curate and maintain.

Vrandečić:As far as I can tell, the problem with the Cebuano Wikipedia is not the availability of content to readers of Cebuano – but that the Cebuano community has no control over this content, how much of it the community wants to integrate, how to filter the machine-generated content out, and how to change it – in short, how to take ownership of the content. These articles are not really owned and maintained by the Cebuano community, but by a single person, who, in dialog with the community, runs his bot to write these millions of articles.

But there is no proper path for the wider community to take ownership of this large set of articles. This is one thing Abstract Wikipedia aims to change: the content and the integration in the local communities will be entirely controlled by the local community. Not only whether they want to pull in Abstract Wikipedia wholesale or not, but in much more detail. And the community members will always be able to go to Abstract Wikipedia and work on the content itself. They are not just passive recipients of content, but they can make their voices heard and join in the creation and maintenance of the content across languages.

I agree with the sentiment of your question, and I do have a similar worry: it is imperative that Abstract Wikipedia does not become another tool for colonizing the knowledge sphere even more thoroughly. Even with the Cebuano community members contributing to Abstract Wikipedia, they will likely be outnumbered by community members from Europe and the US. What are your ideas that, if implemented, could help with Abstract Wikipedia becoming a true tool towards knowledge equity?

Ford: You’re right that the problem with Cebuano Wikipedia is not the availability of content but in the control over that content. What is interesting is that it was a Cebuano Wikipedian who built the first bot that “dump(ed) articles on all the communes of France (over 50,000) on Cebuano Wikipedia,” according to long-time Filipino Wikipedian Josh Lim. 

Lim wrote that the result wasn’t only that “local content was drowned out,” but that it sparked “a numbers war among local Wikipedias” when “other Philippine-language Wikipedias tried to outdo one another in achieving high article counts.” The move to automate articles brought with it a set of values that prioritised quantity over quality, and prioritised subjects that could easily be automatically added (communes of France, for example) over locally relevant topics. It affected the ability of the small Cebuano community to grow its content organically and caused embarrassment when Filipino Wikipedians met other Wikipedians abroad as Cebuano Wikipedia was nicknamed “the Wikipedia of French communes.”

The answer, I think, is for us to think more carefully about what we mean when we say that the “community” will be able to “control” their Abstract Wikipedia project. What defines the Cebuano Abstract Wikipedia community? And how will they make decisions about the content in their repository? The problem with Cebuano Wikipedia was that concerned community members only realised the scale of the problem after the first bot had done its work and the editor running the bot had left the project. There was no consultation. And after the precedent had been set, editors consented to further bot work because they were in a race to increase the visibility of their Wikipedia project. The problem wasn’t only about the domination of Western knowledge. It was about the domination of a logic that recognises the number of articles as a heuristic for quality and/or as a way that small projects gain visibility on the project. 

Is there a way that we can, at the outset of a project, think about progress beyond quantitative heuristics towards qualitative ones? This could mean focusing not on the numbers of articles created but on the development of local principles and policies, for example. 

Can we start small and focus on inclusive deliberation with language communities beyond Wikimedia? This could mean working with only a few languages during an evaluation period in which there is a focus on changing the scope and venue for deliberation – beyond discussions with the few who have the interest, knowledge, and time to give their opinion on wiki, to equipping representatives from the language community with the knowledge necessary for them to make an informed, collective decision beyond the wiki. 

Can we provide communities with the real ability to reject the project and at different stages, using the same principles of deliberation? This won’t be as easy as how most projects are launched and conducted, but it will reap returns because it will foreground planning and discussion and lead to a decision that can be ethically justified. 

Finally, can we invite social scientists working with interested Wikimedians to study the project as it evolves, suggesting important research questions that you highlight here so that we can evaluate the extent to which Abstract Wikipedia fulfills the goals it seeks over time? A number of principles and evaluation frameworks are emerging to investigate the ethical impact of automated systems. There are some very promising evaluation frameworks and principles emerging from this research and it makes sense that Abstract Wikipedia engage with them.   

Vrandečić: I would love for us to move from a metric of number of articles to a more meaningful metric, such as the number of contributors. I think this would be a much more meaningful metric reflecting the health of our projects.

I am wary of setting up deliberative bodies that stretch beyond Wikimedia to make decisions about the Wikimedia projects. One of our strengths is the autonomy of the projects. Setting up a body that is above the actual productive community working on the wiki and possibly dictating which kind of content should and should not be written sounds incompatible with the way our projects have worked so far. Inviting new people into the Wikimedia projects, yes, absolutely, but I think that in the end what happens in a Wikipedia language edition should be decided by the active community members of that given Wikipedia.

What I can commit to is to set up a forum to explicitly discuss the ethics and the implications of the Abstract Wikipedia project, and to send out a wider invitation beyond the current Wikimedia communities. This should allow us to identify potential problems and to either design solutions or avoid parts of the project altogether. One result would indeed be to make sure that the local communities retain control of the whole process of integrating content from Abstract Wikipedia, not just nominally but also practically and deliberately. I would be honored if you’d join us in this forum.

Ford: I get what you’re saying about the problems of non-Wikimedia editors making decisions that Wikimedians have to implement. But I believe that it is a mistake to limit deliberation about the future of Wikimedia projects to the wiki. It’s a mistake because what is presented as fact on Wikimedia isn’t just another representation acting as an equal player in a sea of alternative representations. Wikipedia and the Wikimedia projects that serve it are a dominant representation that affect people well beyond the wiki and around the world because it is increasingly recognised as a global consensus view about people, places, events and things around the world. That means that there are stakeholders well beyond those represented by a small Wikipedia community who should be a part of those discussions because it is those communities who will be affected by those representations. Expanding those discussions doesn’t necessarily mean that the result will be decisions that Wikimedians can’t implement. Recent innovations in deliberation design have demonstrated how this is, indeed, possible to achieve. I am really pleased that you will set up a forum to discuss the ethics and implications of the Abstract Wikipedia project and I look forward to finding ways to expand the deliberative scope of that conversation. Thank you for your invitation! 

-----

FN1. Abstract Wikipedia is a provisional name for this new project and the selection of a final name is still underway.

---

Denny Vrandečić is a computer scientist who founded Wikidata in 2012 so that information could be shared across projects and languages. This year, Vrandečić followed up with Abstract Wikipedia for more complex expressions (note Abstract Wikipedia is a provisional name for this new project and the selection of a final name is still underway). Consider the statement: “Marie Curie is the only person who received two Nobel Prizes in two different sciences.” In his contribution to the book, he describes how such a claim might be represented in an abstract and computer-understandable way and then automatically translated across languages.

Heather Ford is a scholar of Wikipedia and Internet governance and author of the chapter Rise of the Underdog. In her forthcoming book about the Wikipedia “fact factory,” Ford traces the development of the 2011 Egyptian Revolution article to show how factual claims are conceived, contested, and constructed at Wikipedia before being used by the likes of Google and Bing. It is a story about how knowledge automation offers great efficiencies while challenging some of the ideals of the Internet as global public infrastructure. 

For further reading on linked data and automation in the context of Wikipedia and Google, see Heather Ford’s 2016 work in collaboration with Mark Graham from the Oxford Internet Institute in a chapter for the book Code and the City edited by Kitchin and Perng called “Semantic Cities: Coded Geopolitics and the Rise of the Semantic Web” (pre-print), and a journal article for Environment and Planning D “Provenance, Power and Place: Linked Data and Opaque Digital Geographies” (pre-print). 

The Sale of Heritage on eBay: Market Trends and Cultural Value

by Mark Altaweel and Tasoula Georgiou Hadjitofi

Big Data & Society 7(2), DOI https://doi.org/10.1177/2053951720968865. First published: Nov 11th, 2020.

Understanding the heritage market, which includes sales of portable objects from the ancient to more recent past, has been a challenge for heritage experts. In part, this is because the market is not only dynamic but has also included illegal sales that are difficult to track. This study attempts to understand the heritage market by using a potentially important proxy, specifically eBay’s site for selling heritage on the international market. By looking at eBay, the relatively large volume of sales can be monitored to determine patterns of sales that include what cultures are sold, the types of objects sold, and the materials that they are made from. Additionally, it is possible to determine where items are sold from, including what country are selling given objects. While sales data provide us with potential insights that may inform on the wider market, what is also useful is we can use a site like eBay to see how clearly cultural value may potentially drive sales. Cultural value is defined as the appreciation or beliefs one develops about given cultures, including ancient ones, from media, educational, and other resources. Effectively cultural value is driven by our experiences in our modern lives that make us value given cultures, including ones from the ancient past, more than others. The link between cultural value and monetary value is potentially a strong link, as the results of this work show.

Data on eBay sales are partially unstructured. This requires an approach that not only can obtain relevant data but can apply machine learning methods to determine relevant terms for describing information about cultures, types of items, and materials for given objects sold. Named entity recognition (NER) has been a category of methods that attempts to meaningfully categorise terms that could be summarised for information, including quantitative methods that summarise wider term patterns. Conditional random fields (CRFs) are one way to apply NER, where word patterns can be recreated as an undirected graph and the probabilities of terms associated with other terms can help to categorize a given term. For instance, Iron Age can be categorised as a period, based on the association of Age with Iron. However, iron in the terms iron sword can be categorised as a metal given its association with sword. These term associations help to categorised such unstructured descriptive terms that are provided as part of the sale data on eBay. Additionally, using term dictionaries can assist with the fact that sometimes the lack of training data may make categorisation more difficult. Overall, the approach applied in this work allows us to relatively accurately categorise the cultures, types of objects, and the materials objects are made from. Structured data within eBay also allow us to know the countries where sales happen, the price items sold at, and even the sellers (which was anonymised in the article but tracked for statistical purposes).

For the study period (October 21, 2018-January 3, 2020), countries such as the UK, US, Cyprus, Germany, and Egypt are among leading sellers of antiquities, or heritage objects, on eBay. These countries have shown strong cultural value for the cultures that appear to sell the most on eBay. Cultures such as the Romans, ancient Egypt, Vikings (or Norse/Danes), and the ancient Near East have been a fixture in Western education and media and are the top selling cultures. Additionally, items such as jewellery, statues and figurines, and religious items sold the most; masks and vessels (e.g., such as vases) may not have as high a volume in sales but generally fetch higher prices. Metal, stone, and terracotta are also the most commonly sold materials. On the other hand, ivory, papyrus, and wood obtain some of the higher prices. What is also clear is that sales are often driven by a relatively small number of sellers, with about 40% of sales over part of the study period dominated by 10 sellers. Sales disproportionally concentrate in Western countries, but emerging markets such as Thailand are evident. Many countries that dominate sales in one category of items also dominate sales for other objects (Figure 1). Some countries, however, are also specialists with certain cultures or object types, such as Canada, Latvia, India, and Israel. The key finding in this work is we can see that cultural value has a link with sales on eBay and eBay, at least in part, can act as a proxy for wider antiquity sales given that it seems to demonstrate Western markets dominate sales, similar to what has been noticed anecdotally. Other sites and even social media, in the long term, would need to be monitored for their heritage sales to obtain a fuller idea of the market. This work is a start, with the code and data used provided as part of the outputs.

Figure 1. Maps showing where cultures were sold (a) and total sales for countries (USD).

Techno-solutionism and the standard human in the making of the COVID-19 pandemic

Stefania Milan introduces her commentary "Techno-solutionism and the standard human in the making of the COVID-19 pandemic" in Big Data & Society 7(2), https://doi.org/10.1177/2053951720966781. First published: October 20, 2020.

Video abstract

Text abstract

Quantification is particularly seductive in times of global uncertainty. Not surprisingly, numbers, indicators, categorizations, and comparisons are central to governmental and popular response to the COVID-19 pandemic. This essay draws insights from critical data studies, sociology of quantification and decolonial thinking, with occasional excursion into the biomedical domain, to investigate the role and social consequences of counting broadly defined as a way of knowing about the virus. It takes a critical look at two domains of human activity that play a central role in the fight against the virus outbreak, namely medical sciences and technological innovation. It analyzes their efforts to craft solutions for their user base and explores the unwanted social costs of these operations. The essay argues that the over-reliance of biomedical research on “whiteness” for lab testing and the techno-solutionism of the consumer infrastructure devised to curb the social costs of the pandemic are rooted in a distorted idea of a “standard human” based on a partial and exclusive vision of society and its components, which tends to overlook alterity and inequality. It contends that to design our way out of the pandemic, we ought to make space for distinct ways of being and knowing, acknowledging plurality and thinking in terms of social relations, alterity, and interdependence.

Keywords: COVID-19, calculation, whiteness, contact tracing, decolonial, pluriverse

The Norwegian Covid-19 tracing app experiment revisited

by Kristin B Sandvik

In my Big Data & Society commentary ‘Smittestopp: If you want your freedom back download now’ (published on July 28, 2020 within the Viral Data symposium) I mapped out the first phase of the Norwegian version of a Covid-19 tracing app. My goal in the commentary was to “engage critically with the Smittestopp app as a specifically Norwegian technofix ... co-created by the mobilization of trust and dugnaðr, resulting in the launch of an incomplete and poorly defined data-hoarding product with significant vulnerabilities.” Since I submitted my final version of the commentary the app went down in flames and a rancorous domestic blame game ensued, only for the app to be rescheduled for a phoenix-like rebirth in late 2020.

In this blog post, I want to contemplate a set of issues pertaining specifically to the legacy of Smittestopp, but of relevance to other Covid-19 tracing apps. This relates to how democratic government actors respond to criticism of digital initiatives in the context of emergencies; and the type of challenges civil society actors face in holding public and private sector actors accountable. For context, I begin by giving a recap (a longer version here) of the rise and fall of Smittestopp. All translations from Norwegian are my own.

The rise and fall of Smittestopp

April 16, 2020, the Norwegian COVID-19 tracking app Smittestopp was launched to great fanfare. The Ministry of Health, the Norwegian Institute of Public Health (NIPH), the developer of Smittestopp, Simula – a government founded research laboratory (and thus a public entity) – and the prime minister, Erna Solberg, underscored the civic duty to download the app. Norway’s total population is 5.3 million. At the time of the launch, “enough people” was thought to be 50–60% of the population over the age of 16. At its height, the app had almost 900 000 active users. By the end in early June, it had been downloaded 1.6 million times.

Starting weeks before the actual launch of the app in Mid-April, the Norwegian Data protection Authorities (DPA), part of the media, some politicians and a large slice of the domestic tech community had grown exasperated with the procurement processes, functionalities, and designated application of Smittestopp. From the launch date, the app was hampered by technical problems and subjected to a deluge of criticism from self-organizing members of the Norwegian tech and data protection community for being experimental, intrusive, non-transparent, relying on unfounded techno-optimism and abusing trust. As succinctly phrased by a tech activist, the grievance was that NIPH /Simula did not listen to experts, and “made a surveillance app with infection tracing, not an infection tracing app”.

By June 12, DPA issued a formal warning to the NIPH that they were considering a temporary ban on the app due to the app being disproportionately intrusive with respect to personal data. By June 15, the NIPH stopped collecting data, declaring that this meant a weakness of preparedness against increased infection rates as we are ‘loosing time to develop and test the app’, and at the same time loosing our capability to fight the spread of COVID-19. By June 16, Amnesty published a report declaring that “Bahrain, Kuwait and Norway have run roughshod over people’s privacy, with highly invasive surveillance tools which go far beyond what is justified in efforts to tackle COVID-19”. On July 6, the DPA issued a temporary ban on the app. September 28 it was finally over. The Minister of Health, Bent Høie, clarified that 40 million Norwegian kroner of the taxpayer money and months of work had to be scrapped and that a new app would only be ready by Christmas.

Dealing with criticism while Norwegian

While the heated online and offline exchanges over why Smittestopp failed and who was to blame are best described as mudslinging, they also shed light on how Norwegian government actors – rightly praised for the Covid-19 response and generally unaccustomed to international condemnation – respond to criticism of digital initiatives in the context of emergencies. After the cancellation of Smittestopp, Simula launched a determined effort to manage the post-Smittestopp narrative, through the publication of a document dedicated to absolving themselves of most criticism, and forceful media engagement, including friendly reporting and less friendly tirades at twitter and elsewhere about the flaws of both criticisms and critics. Similarly, the health authorities have also been curiously unwilling to accept the legitimacy of the criticism of the legality of the app and the reasons for why it was banned. The storyline projected in these engagements is worth summarizing.

Blaming Big Tech. According to what can best be described as a ‘blaming Big Tech narrative’, Smittestopp was on a path to become a feat of digital innovation but was undermined by opposition from Google and Apple, effectively rendering nation states helpless in the face of the tactics and strategies of Big Tech. Simula contends that “The unique experience of developing Smittestopp in collaboration with the authorities in other countries and Europe's top technologists has been a reminder of how helpless national states can become in the face of global IT companies”, with reference to a critical report (here) on the privacy problems with Google Play from Trinity College. It’s vital for nations to regulate Big Tech – but the lack of adequate regulation was not why Smittestopp failed.

Blaming Amnesty. For a country that prides itself of being a humanitarian superpower and human rights trailblazer, the domestic human rights sector has been curiously silent with respect to the governments Covid-19 efforts, and Smittestopp in particular. When international actors entered the ranks of the critics, further acrimony ensued. Simula has tussled with Amnesty international, suggesting that the organization was not up front about its criticism when the organization initially made contact with the government and called the report from June for “absolute trash”, stating that it’s a report of “exceptionally low quality”, and that Amnesty is abusing their position and power, providing poorly documented conclusions and behaving in an unprofessional manner, letting itself be abused by activists (!), stating that “the meaninglessness of their claims is high as the sky” (the audio file in Norwegian is here).

Blaming the DPA. In radio and television appearances by NIPH and the Minister of Health following the formal cancellation of Smittestopp, the minister of health blamed the DPA for the Smittestopp fiasco arguing that the solution being worked on in June would have effective and complied with data protection regulations. This also meant that the health authorities did not get access to important COVID-19 tracing data. The Minister and NIPH furthermore disagreed that the app was illegal and would like “clearer advice” in the future. The head of the DPA replied that it was not a lack of advice but things the stakeholders themselves did as well as the inability to document the utility of the app which made it disproportionately intrusive and thus illegal- and that the law applies, also in times of national emergencies.

The future of civic activism in emergencies

The passion with which activists, technologists, bureaucrats and scholars have fought against the app, and the strongly worded accusations flying from both sides also suggest that for the rule of law and democratic accountability, struggles over personal data and privacy will continue to get fiercer. The lack of interest in adopting a lessons learned perspective, the absence of humility, the governments continued and willful misunderstanding of how GDPR – and the DPA – works, and the branding of critics as selfish, abusive and cowardly have been an extraordinary spectacle to watch. As observed by a tech activist:

“We have a high level of trust in the state and the government – because the government rarely demands more of its citizenry than is deemed reasonable. Simula, on the other hand, acts if this trust gives us the opportunity to intervene into the private sphere in ways we would call scary if undertaken by a country like Russia or even Great Britain.”

However, an important takeaway from this controversy is what this suggests about the future of civic activism in a country like Norway, which is not historically home to strong foundations, labs, non-profits or grassroots NGOs focusing on holding actors democratically accountable in the digital domain. From early March, individual data activists have been using the Freedom of Information Act to painstakingly map government, Health authorities and DPA interaction on Smittestopp. The focus has subsequently turned to efforts to get access to code and to data sets. At present, work is ongoing to clarify what exactly is going on with the remnants of the Smittestopp infrastructure. This attention to the zombification of data structures post-app is important: Across the globe, Covid-19 apps have collected an enormous amount of data. Governments must be held accountable for mission and function creeps – and for what happens when the apps are set on the path to digital graveyards. This creates questions about what new types of challenges to competencies and methods the established civil society and academia are facing and should be ready to meet.

Smittestopp is dead! Long live Smittestopp!

What have we learned from all of this? While Smittestopp is small fry, it is also an instructive illustration of the affordances of the crisis-label and the pitfalls of digital transformation in a highly digitized society. The experiment now continues. The preferred solution for the new Norwegian app is GAEN (Google and Apple Exposure Notification), with data stored mostly locally. Possibly due to the public brouhaha, the tender process only received one offer – which was accepted – from the Danish NetCompany, the developer of the Danish Covid-19 tracing app also called ‘Smittestopp’. After a public vote of sorts, it is clear that the second installment of the Norwegian app-experiment will also be called Smittestopp. The regulations for the initial app were repealed October 9 and no regulations have been issued for the new app. Netcompany has emphasized the importance of better public communication – but as this blog has illustrated, public engagement on the legality and legitimacy of digital interventions and figuring out the how-to of this – remains more important than ever.

Revisiting the Black Box Society by Rethinking the Political Economy of Big Data

Special Theme Issue
https://journals.sagepub.com/page/bds/collections/revisitingtheblackboxsociety

Guest lead editors: Benedetta Brevini* and Frank Pasquale**

* University of Sydney
** Brooklyn Law School

Throughout the 2010s, scholars explored the politics and sociology of data, its regulation and its role in informing and guiding policymakers such as the importance of quality health data in the COVID-19 epidemic to “flatten the curve.” However, all too much of this work is being done in “black box societies” jurisdictions where the analysis and use of data is opaque, unverifiable, and unchallengeable. As a result, far too often data are used as a tool for social, political, and economic control, with biases often distorting decision making and accompanied by narratives of tech solutionism and even salvation-ism abound.

The Black Box Society was one of first scholarly accounts of algorithmic decision making to synthesize empirical research, normative frameworks, and legal argument and this symposium of commentaries reflect on what has happened since its publication. Much has happened since 2015 that vindicates and challenges the book’s main themes. Yet recurring examples of algorithmically driven injustices raise the question of whether transparency—the foundational normative value in The Black Box Society—is a first step toward a more emancipatory deployment of algorithms and AI, is an easily deflected demand, or actually worsens matters by rationalizing the algorithmic ordering of human affairs.

To address these issues, this symposium features the work of leading thinkers who have explored the interplay of politics, economics, and culture in domains ordered algorithmically by managers, bureaucrats, and technology workers. By bringing social scientists and legal experts into dialogue, we aim both to clarify the theoretical foundations of critical algorithm studies and to highlight the importance of engaged scholarship, which translates the insights of the academy into an emancipatory agenda for law and policy reform. While the contributions are diverse, a unifying theme animates them: each offers a sophisticated critique of the interplay between state and market forces in building or eroding the many layers of our common lives, as well as the kaleidoscopic privatization of spheres of reputation, search, and finance. Unsatisfied with narrow methodologies of economics or political science, they advance politico-economic analysis. They therefore succeed in unveiling the foundational role that the turn to big data has in organising economic and social relations. All the contributors help us imagine practical changes to prevailing structures that will advance social and economic justice, mutual understanding, and ecological sustainability. For this and much else, we are deeply grateful for their insightful work.

Editorial by Benedetta Brevini and Frank Pasquale, "Revisiting the Black Box Society by rethinking the political economy of big data". 

Ifeoma Ajunwa, in “The Black Box at Work,” describes the data revolution of the workplace, which simultaneously demands workers surrender intimate data and then prevents them from reviewing how it is used.

Mark Andrejevic, in “Shareable and Un-Shareable Knowledge,” focuses on what it means to generate actionable but non-shareable information, reaffirming the urgency of intelligible evaluation as a form of dignity.

Margaret Hu’s article “Cambridge Analytica’s Black Box” surveys a range of legal and policy remedies that have been proposed to better protect consumer data and informational privacy.

Paul Prinsloo examines “Black Boxes and Algorithmic Decision-making in (Higher) Education” to show how the education sector is beginning to adopt technologies of monitoring and personalization that are similar to the way the automated public sphere serves political information to voters.

Benedetta Brevini, in “Black Boxes, not Green: Mythologizing AI and Omitting the Environment” documents how AI runs on technology, machines and infrastructures that deplete scarce resources in their production, consumption and disposal, thus placing escalating demands on energy and accelerating the climate emergency.

Gavin Smith develops the concept of our “right to the face” in “The Face is the Message: Theorisingthe Politics of Algorithmic Governance in the Black Box City” as he explores how algorithms are now responsible for important surveillance of cities, constantly passing judgment on mundane activities.

Nicole Dewandre’s article, “Big Data: From Fears of the Modern to Wake-up Call for a New Beginning” applies a deeply nuanced critique of modernity to algorithmic societies arguing that Big Data may be hailed as the endpoint or materialisation of a Western modernity, or as a wake-up call for a new beginning.

Jonathan Obar confirms this problem empirically in “Sunlight Alone is Not a Disinfectant: Consent andthe Futility of Opening Big Data Black Boxes,” and proposes solutions to more equitably share the burden of understanding.

Kamel Ajji in “CyborgFinance Mirrors Cyborg Social Media” outlines how The Black Box Society inspired him to found “21 Mirrors, a nonprofit organization aimed at analyzing, rating and reporting to the public about the policies and practices of social media, web browsers and email services regarding their actual and potential consequences on freedom of expression, privacy, and due process.”