AI audits for assessing design logics and building ethical systems: the case of predictive policing algorithms

Organisations, governments, institutions and others across several jurisdictions are using AI systems for a constellation of high-stakes decisions that pose implications for human rights and civil liberties. But a fast-growing multidisciplinary scholarship on AI bias is currently documenting problems such as the discriminatory labelling and surveillance of historically marginalised subgroups. One of the ways in which AI systems generate such downstream outcomes is through their inputs. This paper focuses on a specific input dynamic which is the theoretical foundation that informs the design, operation, and outputs of such systems. The paper uses the set of technologies known as predictive policing algorithms as a case example to illustrate how theoretical assumptions can pose adverse social consequences and should therefore be systematically evaluated during audits if the objective is to detect unknown risks, avoid AI harms, and build ethical systems. In its analysis of these issues, the paper adds a new dimension to the literature on AI ethics and audits by investigating algorithmic impact in the context of underpinning theory. In doing so, the paper provides insights that can usefully inform auditing policy and practice instituted by relevant stakeholders including the developers, vendors, and procurers of AI systems as well as independent auditors.

Ten-Four

This story is an appropriation/erasure of George Orwell’s 1984, remixing some of its original text with concepts from popular fiction and academic literature, including my published work. These concepts include: lateral surveillance; pandemic policing; data capitalism; predictive policing; and posthuman. The story contributes to the diverse, expanding field of cultural production known as “speculative fiction… a mode of thought-experimenting” (Oziewicz 2017). Rather than trying to predict the future, speculative fiction “unsettles the present” with what-if questions that allow us to “develop alternative social imaginaries and open up new perspectives” and “spaces of debate” (Dunne and Raby 2013: chapters 88, 189, 3).

Privatization of Law Enforcement

The year 2020 ushered in growing calls to defund the police. In Canada, as in other countries where the movement to defund the police has gained momentum, activists demand transferring money from police departments to social workers, reducing the number of police officers, and increasing police departments’ democratic accountability. This last group of reform initiatives is, perhaps, the least controversial one because it calls for improving the familiar structures of democratic oversight over police departments, such as municipal councils, independent police oversight boards and complaints bodies, and others. The demands for greater accountability of police departments to the public are a symptom of a deeper problem - there is a growing discrepancy between the goals of policing and the consequences of the police’s actions. This discrepancy materializes when the police’s attempts to ensure public safety result in the marginalization of racialized communities, particularly in larger cities across Canada. In order to understand why laudable policy goals lead to deeply problematic consequences, it is necessary to analyze the policing process in our cities. While it has traditionally been assumed that this process is left to the discretion of separate police departments, this paper demonstrates that externalities, such as data generated by private technologies, play an important role in undermining the goals of policing. Reliance on private data and technology does not absolve the police of accountability for resulting human rights violations. However, it has important implications for the reform of public oversight over the police. In an era when non-governmental actors are taking part in law enforcement through procurement contracts, democratic control over the exercise of the police’s contracting powers is an important, albeit often overlooked, instrument of police reform. Relying on contracts for predictive policing technologies as a case study, this paper argues that communities should condition the funding of police procurement on ex ante assessment procedures, technical specifications, and contract enforcement rights. Also, local elected representatives should have an opportunity to approve any data and technology sharing arrangements as well as federal standing offer arrangements that extend predictive policing to their communities.

Predictive Policing

This chapter addresses predictive policing, which is a term that refers to a range of crime-fighting approaches that use crime mapping data and analysis, and, more recently, social network analysis, big data, and predictive algorithms. The rise of predictive policing, especially in many police jurisdictions in large cities in the USA, has raised the spectre of the surveillance society in which citizens can be arrested by police for crimes they have not yet committed on the basis of evidence that they will commit them. Speaking generally, predictive policing faces several problems. Some of these are problems for predictive policing even in its own terms of contributing to crime reduction. Others are moral problems, about whether predictive policing violates moral rights or is unjust. These two types of problems are interconnected. Ultimately, the expanding use of biometric facial recognition databases and other emerging technologies in law enforcement as part of predictive policing should be clearly and demonstrably justified in terms of efficiency and effectiveness in the service of specific law enforcement purposes rather than by general appeals to community security or safety. Moreover, it should comply with moral principles constitutive of liberal democracy, such as the principle that individuals have a moral right to freedom from state interference absent prior evidence of violation of its laws.