Vice reports on a Tokyo-based company, DeepScore, pitching software for the automatic recognition of ‘trustworthiness’, e.g. in loan applicants. Although their claimed false-negative rate of 30% may not sound particularly impressive, it must of course be compared to well-known human biases in lending decisions. Perhaps more interesting is the instrumentalization cycle, which is all but assured to take place if DeepScore’s algorithm gains wide acceptance. On the one hand, the algorithm’s goal is to create a precise definition for a broad and vague human characteristic like trustworthiness—that is to say, to operationalize it. Then, if the algorithm is successful on its training sample and becomes adopted by real-world decision-makers, the social power of the adopters reifies the research hypothesis: trustworthiness becomes what the algorithm says it is (because money talks). Thus, the behavioral redefinition of a folk psychology concept comes to fruition. On the other hand, however, instrumentalization immediately kicks in, as users attempt to game the operationalized definition, by managing to present the algorithmically-approved symptoms without the underlying condition (sincerity). Hence, the signal loses strength, and the cycle completes. The fact that DeepScore’s trustworthiness algorithm is intended for credit markets in South-East Asia, where there exist populations without access to traditional credit-scoring channels, merely clarifies the ‘predatory inclusion’ logic of such practices (v. supra).
Tag Archives: Readings
Trustworthiness of unfree code
Several reports are circulating (e.g., via /.) of a court case in New Jersey in which the defendant won the right to audit proprietary genetic testing software for errors or potential sources of bias. It being a murder trial, this is about as close to a life-or-death use-case as possible.
Given the stakes, it is understandable that a low-trust standard should prevail in forensic matters, rendering an audit indispensable (nor is the firm’s “complexity defence” anything short of untenable). What is surprising, rather, is how long it took to obtain this type of judicial precedent. The authoritativeness deficit of algorithms is a topic of burning intensity generally; that in such a failure-critical area a business model based on proprietary secrecy has managed to survive is truly remarkable. It is safe to say that this challenge will hardly be the last. Ultimately, freely auditable software would seem to be the superior systemic answer for this type of applications.
Lye machines
Josephine Wolff (Slate) reports on the recent hack of the water processing plant in Oldsmar, FL. Unknown intruders remotely accessed the plant’s controls and attempted to increase the lye content of the town’s water supply to potentially lethal levels. The case is notable in that the human fail-safe (the plant operator on duty) successfully counterbalanced the machine vulnerability, catching the hack as it was taking place and overriding the automatic controls, so no real-world adverse effects ultimately occurred.
What moral can be drawn? It is reasonable to argue, as Wolff does, against full automation: human supervision still has a critical role to play in the resiliency of critical control systems through human-machine redundancy. However, what Wolff does not mention is that this modus operandi may itself be interpreted as a signature of sorts (although no attribution has appeared in the press so far): it speaks of amateurism or of a proof-of-concept stunt; in any case, of an actor not planning to do any serious damage. Otherwise, it is highly improbable that there would have been no parallel attempt at social engineering of (or other types of attacks against) on-site technicians. After all, as the old security engineering nostrum states, rookies target technology, pros target people.
Media manipulation convergence
Adam Satariano in the NYT reports on the latest instance of platform manipulation, this time by Chinese tech giant Huawei against unfavorable 5G legislation being considered in Belgium. There’s nothing particularly novel about the single pieces of the process: paid expert endorsement, amplified on social media by coordinated fake profiles, with the resultant appearance of virality adduced by the company as a sign of support in public opinion at large. If anything, it appears to have been rather crudely executed, leading to a fairly easy discovery by Graphika: from a pure PR cost-benefit standpoint, the blowback from the unmasking of this operation did much more damage to Huawei’s image than any benefit that might have accrued to the company had it not been exposed. However, the main take-away from the story is the adding of yet another data point to the process of convergence between traditional government-sponsored influence operations and corporate astroturfing ventures. Their questionable effectiveness notwithstanding, these sorts of interventions are becoming default, mainstream tools in the arsenal of all PR shops, whatever their principals’ aims. The fact that they also tend to erode an already fragile base of public trust suggests that at the aggregate level this may be a negative-sum game.
Reddit mobs rampaging on the stockmarket
I am following (just like everyone else) the developing GameStop story. Beyond the financial technicalities, what is interesting for present purposes is that the dynamics of internet virality seem to be finding a close parallel in stock valuation. The term “meme stock” is telling. In other words, at present the online coordination mechanisms, the capital, and the nihilistic boredom are all available to craft an alternative description of reality, which in turn is self-reinforcing (until it isn’t).