Tag Archives: Risk

Organizational behavior and reputation

Public red-teaming and trust

DEF CON is one of the most important hacker conferences worldwide, held yearly in Las Vegas. This coming August, it will host a large simulation, in which thousands of security experts from the private sector and academia will be invited to compete against each other to uncover flaws and bias in the generative large language models (LLMs) produced by leading firms such as OpenAI, Google, Anthropic, Hugging Face, and Stability. While in traditional red-team events the targets are bugs in the code, hardware, or human infrastructure, participants at DEF CON have additionally been instructed to seek exploits through adversarial prompt engineering, so as to induce the LLMs to return troubling, dangerous, or unlawful content.

This initiative definitely goes in the right direction in terms of building trust through verification, and bespeaks significant confidence on the part of the companies, as it can safely be expected that the media outlets in attendance will be primed to amplify any failure or embarassing shortcoming in the models’ output. There are limits, however, to how beneficial such an exercise can be. For one thing, the target constituency is limited to the extremely digitally literate (and by extension to the government agencies and private businesses the firms aspire to add to their customer list): the simulation’s outcome cannot be expected to move the needle on the broad, non-specialist perception of AI models and their risks in the public at large. Also, the stress test will be performed on customized versions of the LLMs, made available by the companies specifically for this event. The Volkswagen emissions scandal is only the most visible instance of how one may exploit such a benchmarking system. What is properly needed is the possibility of an unannounced audit of LLMs on the ground in their actual real-world applications, on the model of the Michelin Guide’s evaluation process for chefs and restaurants.

In spite of these limitations, the organization of the DEF CON simulation if nothing else proves that the leading AI developers have understood that wide-scale adoption of their technology will require a protracted engagement with public opinion in order to address doubts and respond to deeply entrenched misgivings.

Trustworthiness

Societal trust and the pace of AI research

An open letter from the Future of Life Institute exhorts the leading AI labs to enact a six-month moratorium on further experiments with artificial intelligence. The caliber of some of the early signatories guarantees that significant public conversation will ensue. Beyond the predictable hype, it is worth considering this intervention in the AI ethics and politics debate both on its merits and for what it portends more broadly for the field.

First off, the technicalities. The text locates the key chokepoint in AI development to be exploited in the interests of the moratorium in the scarcity of compute power. Truly, we are at the antipodes of the decentralized mode of innovation that drove, for instance, the original development of the commercial and personal web in the 1990s. However, it remains to be seen whether the compute power barrier has winnowed down the field into enough of an oligopoly for the proposed moratorium to have any chance of application. A closely related point is verifiability: even if there were few enough players to enable a coordination regime to emerge and there was virtually universal buy-in, it would still be necessary to enact some form of verification in order to police the system and ensure nobody is cheating. By comparison, the nuclear non-proliferation regime enjoys vast buy-in and plentiful dedicated enforcement resources (both at the nation-state and at the international organization level) and yet is far from perfect or fool-proof.

Moving to broader strategic issues, it bears considering whether the proposed moratorium, which would necessarily have to be global in scope, is in any way feasible in the current geopolitical climate. After all, one of the classic formulations of technological determinism relies on Great Power competition in military and dual-use applications. It would not be outlandish to suggest that we already are in a phase of strategic confrontation, between the United States and China among others, where the speed of tech change has become a dependent variable.

Perhaps, however, it is best to consider the second-order effects of the letter as the crux of the matter. The moratorium is extremely unlikely to come about, and would be highly unwieldy to manage if it did (the tell, perhaps, is the mismatch between the apocalyptic tone in which generative AI is described and the very short time requested to prepare for its onslaught). Nonetheless, such a proposal shifts the debate. It centers AI as the future technology to be grappled with socially, presents it as largely inevitable, and lays the responsibility for dealing with its ills at the foot of society as a whole.

Most strikingly, though, this intervention in public discourse relies on very tenuous legitimacy grounds for the various actors concerned, beginning with the drafters and signatories of the letter. Is the public supposed to endorse their analysis and support their prescriptions on the basis of their technical expertise? Or their impartiality? Or their track record of civic-mindedness? Or their expressing of preferences held by large numbers of people? All these justifications are problematic in their own way. In a low-trust environment, the authoritativeness of a public statement conducted in this fashion is bound to become itself a target of controversy.

Organizational behavior and reputation

Digital Welfare Systems

An extremely interesting series of talks hosted by the Digital Freedom Fund: the automation of welfare system decisons is where the neoliberal agenda and digitalization intersect in the most socially explosive fashion. All six events look good, but I am particularly looking forward to the discussion of the Dutch System Risk Indication (SyRI) scandal on Oct. 27th. More info and free registration on the DFF’s website.

Online sociability and community

Sharp Eyes

An interesting report in Medium (via /.) discusses the PRC’s new pervasive surveillance program, Sharp Eyes. The program, which complements several other mass surveillance initiatives by the Chinese government, such as SkyNet, is aimed especially at rural communities and small towns. With all the caveats related to the fragmentary nature of the information available to outside researchers, it appears that Sharp Eyes’ main characteristic is being community-driven: the feeds from CCTV cameras monitoring public spaces are made accessible to individuals in the community, whether at home from their TVs and monitors or through smartphone apps. Hence, local communities become responsible for monitoring themselves (and providing denunciations of deviants to the authorities).

This outsourcing of social control is clearly a labor-saving initiative, which itself ties in to a long-run, classic theme in Chinese governance. It is not hard to perceive how such a scheme may encourage social homogeneization and irregimentation dynamics, and be especially effective against stigmatized minorities. After all, the entire system of Chinese official surveillance is more or less formally linked to the controversial Social Credit System, a scoring of the population for ideological and financial conformity.

However, I wonder whether a community-driven surveillance program, in rendering society more transparent to itself, does not also potentially offer accountability tools to civil society vis-à-vis the government. After all, complete visibility of public space by all members of society also can mean exposure and documentation of specific public instances of abuse of authority, such as police brutality. Such cases could of course be blacked out of the feeds, but such a heavy-handed tactic would cut into the propaganda value of the transparency initiative and affect public trust in the system. Alternatively, offending material could be removed more seamlessly through deep fake interventions, but the resources necessary for such a level of tampering, including the additional layer of bureaucracy needed to curate live feeds, would seem ultimately self-defeating in terms of the cost-cutting rationale.

In any case, including the monitored public within the monitoring loop (and emphasizing the collective responsibility aspect of the practice over the atomizing, pervasive-suspicion one) promises to create novel practical and theoretical challenges for mass surveillance.

Political economy and visibility

Behavioral redefinition

Vice reports on a Tokyo-based company, DeepScore, pitching software for the automatic recognition of ‘trustworthiness’, e.g. in loan applicants. Although their claimed false-negative rate of 30% may not sound particularly impressive, it must of course be compared to well-known human biases in lending decisions. Perhaps more interesting is the instrumentalization cycle, which is all but assured to take place if DeepScore’s algorithm gains wide acceptance. On the one hand, the algorithm’s goal is to create a precise definition for a broad and vague human characteristic like trustworthiness—that is to say, to operationalize it. Then, if the algorithm is successful on its training sample and becomes adopted by real-world decision-makers, the social power of the adopters reifies the research hypothesis: trustworthiness becomes what the algorithm says it is (because money talks). Thus, the behavioral redefinition of a folk psychology concept comes to fruition. On the other hand, however, instrumentalization immediately kicks in, as users attempt to game the operationalized definition, by managing to present the algorithmically-approved symptoms without the underlying condition (sincerity). Hence, the signal loses strength, and the cycle completes. The fact that DeepScore’s trustworthiness algorithm is intended for credit markets in South-East Asia, where there exist populations without access to traditional credit-scoring channels, merely clarifies the ‘predatory inclusion’ logic of such practices (v. supra).