Vice reports on a Tokyo-based company, DeepScore, pitching software for the automatic recognition of ‘trustworthiness’, e.g. in loan applicants. Although their claimed false-negative rate of 30% may not sound particularly impressive, it must of course be compared to well-known human biases in lending decisions. Perhaps more interesting is the instrumentalization cycle, which is all but assured to take place if DeepScore’s algorithm gains wide acceptance. On the one hand, the algorithm’s goal is to create a precise definition for a broad and vague human characteristic like trustworthiness—that is to say, to operationalize it. Then, if the algorithm is successful on its training sample and becomes adopted by real-world decision-makers, the social power of the adopters reifies the research hypothesis: trustworthiness becomes what the algorithm says it is (because money talks). Thus, the behavioral redefinition of a folk psychology concept comes to fruition. On the other hand, however, instrumentalization immediately kicks in, as users attempt to game the operationalized definition, by managing to present the algorithmically-approved symptoms without the underlying condition (sincerity). Hence, the signal loses strength, and the cycle completes. The fact that DeepScore’s trustworthiness algorithm is intended for credit markets in South-East Asia, where there exist populations without access to traditional credit-scoring channels, merely clarifies the ‘predatory inclusion’ logic of such practices (v. supra).
Given the recent salience of news on surveillance and surveillance capitalism, it is to be expected that there would be rising interest in material, technical countermeasures. Indeed, a cottage industry of surveillance-avoidance gear and gadgetry has sprung up. The reviews of these apparatuses tend to agree that the results they achieve are not great. For one thing, they are typically targeted at one type of surveillance vector at a time, thus requiring a specifically tailored attack model rather than being comprehensive solutions. Moreover, they can really only be fine-tuned properly if they have access to the source code of the algorithm they are trying to beat, or at least can test its response in controlled conditions before facing it in the wild. But of course, uncertainty about the outcomes of surveillance, or indeed about whether it is taking place to begin with, is the heart of the matter.
The creators of these countermeasures themselves, whatever their personal intellectual commitment to privacy and anonymity, hardly follow their own advice in eschewing the visibility the large internet platforms afford. Whether these systems try to beat machine-learning algorithms through data poisoning or adversarial attacks, they tend to be more of a political statement and proof of concept than a workable solution, especially in the long term. In general, even when effective, using these countermeasures is seen as extremely cumbersome and self-penalizing: they can be useful in limited situations for operating in ‘stealth mode’, but cannot be lived in permanently.
If this is the technological state of play, are we destined to a future of much greater personal transparency, or is the notion of hiding undergoing an evolution? Certainly, the momentum behind the diffusion of surveillance techniques such as facial recognition appears massive worldwide. Furthermore, it is no longer merely a question of centralized state agencies: the technology is mature for individual consumers to enact private micro-surveillance. This sea change is certainly prompting shifts in acceptable social behavior. But as to the wider problem of obscurity in our social lives, the strategic response may well lie in a mixture of compartimentalization and hiding in plain sight. And of course systems of any kind are easier to beat when one can target the human agent at the other end.
A new court case has been brought against the City and County of San Francisco for the use of surveillance cameras by the San Francisco Police Department, in violation of a 2019 city ordinance, to control protests in early June following the killing of George Floyd. The Electronic Frontier Foundation and the American Civil Liberties Union of Northern California are representing the three plaintiffs in the suit, community activists who participated in the demonstrations, alleging a chilling effect on freedom of speech and assembly. The surveillance apparatus belonged to a third party, the Union Square Business Improvement Distric, and its use was granted voluntarily to law enforcement, following a request.
Use of surveillance and facial recognition technology is widespread among California law enforcement, but such policies are often opaque and unacknowledged. Police Departments have been able to evade legislative and regulatory curbs on their surveillance activities through third-party arrangements. Such potential for non-compliance strengthens the case for approaches such as that taken in Portland, OR, where facial recognition technology is banned for all, not simply for the public sector.
Just discovered (via the BKC newsletter) a cool publication, Logic. They do three themed issues a year on topics at the intersection of tech and society. Vol. 10: Security (from May this year) looks particularly close to the kind of things I am working on. There’s a long piece by Matt Goerzen and Gabriella Coleman on the intertwined histories of hacking and computer security, and a couple of in-depth interviews with Tawana Petty on facial recognition and with Alison Macrina on Tor. Good stuff: I need to get my hands on a hard copy.