The Global Age of Algorithm - Social Credit and the Financialisation of Governance in China - ANU Press
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
ON A CHINESE SCREEN
‘Face, Face, Face.’ PC:
Terhide Tomori, Flickr.
The Global Age of com.
Algorithm
Social Credit and the
Financialisation of
Governance in China
Nicholas LOUBERE
Stefan BREHM
Much has been made of the Orwellian social Beautiful credit! The foundation of modern
control aspects of the emerging ‘social credit society. Who shall say that this is not the
system’ in China. However, social credit golden age of mutual trust, of unlimited
is more than simply a Chinese version of reliance upon human promises?
big brother: it is an unprecedented climax
of the global financialisation project Mark Twain and Charles Dudley Warner,
and a signal of a potential dark digital The Gilded Age: A Tale of Today (1873)
future dominated by algorithmic rule.
I
n recent years, few news items out of China
have resulted in as much anxiety and fear
in western media and public discourse than
the Chinese government’s ongoing attempts to
create a ‘social credit system’ (shehui xinyong
142 MADE IN CHINA YEARBOOK 2018ON A CHINESE SCREEN
tixi) aimed at rating the trustworthiness big data generated from economic, social,
of individuals and companies. Most major and commercial behaviour. The stated aim
Western media outlets have spent significant is to ‘provide the trustworthy with benefits
energy warning about China’s efforts to create and discipline the untrustworthy … [so that]
an Orwellian dystopia. The most hyperbolic integrity becomes a widespread social value’
of these—The Economist—has even run with (State Council 2016). While official policy
menacing headlines like ‘China Invents documents are light on detail with regard to
the Digital Totalitarian State’ and ‘China’s how the social credit system will ultimately
Digital Dictatorship’ (The Economist 2016b; operate, they have suggested various ways
The Economist 2016c). These articles both to punish untrustworthy members of society
implicitly and explicitly depict social credit as (i.e. those with low ratings), such as through
something unique to China—a nefarious and restrictions on employment, consumption,
perverse digital innovation that could only be travel, and access to credit. In recent months,
conceived of and carried out by a regime like there have already been reports of blacklisting
the Chinese Communist Party (Daum 2017). resulting in restrictions for individuals, but
Social credit is thus seen as signalling the as of yet this only applies to those who have
onset of a dystopian future that could only broken specific laws or ‘failed to perform
exist in the Chinese context. But how unique to certain legal obligations’ (Daum 2018a).
China is this attempt to ‘build an environment The Chinese social credit system is emerging
of trust’—to quote the State Council (2016)— rapidly, and the aforementioned blacklists are
using new digital forms of data collection and connecting data from dozens of governmental
analysis? Is this Orwellian social credit system departments. However, it is still far from
indicative of an inherently Chinese form of being a unified or centralised system. Like
digital life, or is it a dark manifestation of our most new policies in China, social credit is
collective impulses to increase transparency being subjected to the country’s distinctive
and accountability (at the expense of privacy), policy modelling process (Heilmann 2008),
and to integrate everyone into a single where local governments produce their own
‘inclusive’ system to more easily categorise, interpretations of policies, which then vie
monitor, and standardise social activity? In this to become national models. Over 30 local
essay, we propose that Chinese social credit governments have already started piloting
should not be exoticised or viewed in isolation. social credit systems, which utilise different
Rather, it must be understood as merely one approaches to arrive at their social credit
manifestation of the global age of the algorithm. scores, and which use the scores to achieve
different outcomes. In contrast to other
policies, however, large Internet companies
Engineering a have also been given licenses to run their own
Trustworthy Society pilots (Loubere 2017a). The most widely used
private social credit system is Alibaba’s Sesame
Credit, which utilises opaque algorithms
While there have long been discussions to arrive at social credit scores for their
about creating an economic and social rating customers. Those with high scores have been
system in China, they took a much more able to access a range of benefits from other
concrete form in 2014, with the publication of Alibaba businesses and their partners (Bislev
a high-level policy document outlining plans 2017). Sesame Credit is significant due to
to create a nationwide social credit system the huge amounts of economic data held by
by 2020 (State Council 2014). The proposed Alibaba through Alipay and Ant Financial, but
system will assign ratings to individuals, it should not be conflated with governmental
organisations, and businesses that draw on
MADE IN CHINA YEARBOOK 2018 143ON A CHINESE SCREEN
social credit system pilots. It is not clear how inclusion project, which seeks to integrate
or if the government and private systems will marginal and impoverished populations into
be integrated in the future, which seems to be the global capitalist system—primarily through
causing a degree of tension between regulators expanded access to credit—as a means of
and the Internet giants (Hornby, Ju, and promoting economic development and social
Lucas 2018). empowerment.
In the same way that Chinese social credit
appears poised to extract huge amounts of
Financial(ised) Inclusion personal data from individuals in its quest to
create a trustworthy society, proponents of
financial inclusion justify intrusive methods
While social credit can be seen as an of assessing creditworthiness in order to
outgrowth of our collective impulse to achieve reduce lender risk from untrustworthy
a more trustworthy society, a unified fully- borrowers. Indeed, just months before their
functioning social credit system will ultimately hyperbolic headlines about China’s digital
turn the quest for trust through transparency authoritarianism, The Economist praised the
and accountability upside down, because it use of psychometrics and other personal digital
would hold citizens responsible vis-à-vis their data by lenders in developing contexts as being a
rulers. At the core of the emerging system, beneficial financial innovation (The Economist
the state and financial actors define, quantify, 2016a). In this way, the financial inclusion
and calculate trustworthiness and honesty—it project depicts the application of financialised
is a technocratic fix based on the logic that, logics as the means of producing a more fair
with the correct set of algorithms, the good and accountable inclusive system, where the
citizen can be engineered into society. Social trustworthy reap rewards that were denied
credit therefore seeks to transform individuals them in the past. However, underpinning this
into a new ‘civilised’ (and ‘credit-conscious’) neoliberal fantasy is a glaring contradiction
population through the imposition of an that shatters the illusion of inclusion as being
incentive and disincentive system that can unbiased and fair—those with capital are able
mould logical profit-maximising citizens into to set the terms of their engagement with the
civilised subjects. capitalist system much more easily than those
In the case of China’s proposed social credit without.
system—as with any credit rating system— This points to the fact that the rich will
these rewards and punishments are meted out largely be able to extract more of the rewards
through engagement with, and incorporation from their participation in financialised rating
into, the market. The calculation of credit scores systems—such as the social credit system—
requires market activity, which in turn requires while avoiding the sanctions. Moreover,
a credit score. Moreover, if social credit is to live punishments are much more dramatic for
up to its technocratic promise of systematically those without accumulated capital, as their
eliminating untrustworthiness, everyone must very existence depends on their continued
be assessed equally—i.e. everyone must be participation in the capitalist system for daily
included in the system. In the absence of a survival. From this perspective, the spectre
social credit score, the worst must be assumed, of China’s financialised social credit system
meaning that the burden of proving ones portents a society comprising individual micro-
trustworthiness falls to the individual. Thus, in entrepreneurs operating in a shared economy
a society dominated by social credit, integration mode where livelihoods are determined by
into the socioeconomic system is a necessity credit scores. Indeed, Sesame Credit already
rather than a choice. In this way, China’s social works with sharing economy apps, such as
credit resonates with the global financial Daowei, which provides a platform for a literal
144 MADE IN CHINA YEARBOOK 2018ON A CHINESE SCREEN
gig economy comprised of individuals (with maximisation into the realm of interpersonal
their credit scores listed) advertising the sale relationships. Through managing networks,
of their services or products (Loubere 2017b). digital activity, and private action, an individual
Those looking for a plumber in the area can or organisation can impact social value and, by
select one with the highest score, just as people extension, financial capital. Thus, social credit
in the west decide hotels and restaurants based creates new incentives that can be used to
on Yelp or TripAdvisor reviews. align the interests of citizens and organisations
with those of the government. The state,
as a shareholder in ‘the people’, enjoys the
Financialisation Gone dividends of good behaviour and loyalty, which
Wild are rewarded through economic privileges. In
this all-encompassing financialised system,
social action becomes increasingly entrenched
In this sense, the emergence of social credit within the economic realm, and individual
represents an unprecedented climax of the behaviour is more and more shaped by financial
global financialisation project. Financialisation motives. In a nutshell, social credit represents
can be broadly defined as ‘the increasing the ultimate marketisation of political control
role of financial motives, financial markets, because it provides incentives for maximising
financial actors, and financial institutions in citizen value through politically and
the operation of domestic and international commercially aligned social behaviour.
economy’ (Epstein 2005, 3). Social credit opens Using algorithms to render citizens and
the door for financialising social behaviour. organisations compliant with the visions
To elaborate this claim, consider the relation and rationales of the ruling regime reduces
between social and financial capital. The the state’s information and monitoring costs
OECD, for example, defines social capital as dramatically. In the context of China, this
‘networks together with shared norms, values has the potential to reshape the ‘fragmented
and understandings that facilitate cooperation authoritarian’ model, which is characterised
within or among groups’ (Keeley 2007, 103). by decentralised decision-making and policy
In the digital age, these networks become the implementation (Mertha 2009). One could
linchpins between the social and economic envision a future in which the many local
spheres. On the one hand, networks are officials and bureaucrats that enjoy privileges
more concrete and easier to observe than the due to the central leadership’s reliance on their
norms and values shaping the perceptions and support to govern the masses will be subject to
behaviour of network members. On the other the rule of algorithms themselves. Only a small
hand, social networks represent a crucial means elite would be needed to manage algorithmic
for both gaining access to material resources rule, entailing a dramatic reconcentration of
and shaping the rules for resource distribution. power. If Chinese experiments are successful,
Thus, analysing and contextualising social they will certainly serve as a model for many
networks in a big-data driven world allow for other countries: authoritarian regimes,
inferences to be made about both the social and democratic systems with authoritarian
economic attributes of an individual. tendencies, and eventually democracies
The invention of social credit establishes that struggle to maintain legitimacy in an
an explicit and tangible link between social increasingly polarised and fragmented political
behaviour and economic benefits. In this landscape.
context, the State is able to assume a new role
not dissimilar to that of a corporate shareholder.
Social credit creates a market for social
capital and transplants the rationale of profit
MADE IN CHINA YEARBOOK 2018 145ON A CHINESE SCREEN
The Repressive Logics infrastructure primarily targeting the
of Financialised Uyghur ethnic minority. This includes police
checkpoints, iris scans, mandatory spyware
Governance installed on mobile devises, and pervasive
CCTV with facial recognition software. These
surveillance technologies feed into, and draw
As noted above, despite the discourse of on, a database that includes information about
inclusion resulting in transparency and fairer personal identity, family and friends, movement
distribution or resources, social credit and and shopping behaviour, and even DNA that
the financialisation of social behaviour are is collected at medical check-ups organised
inherently biased and paradoxically result by the government. Ultimately, these data are
in socioeconomic exclusion within an all- run through algorithms that assign residents
encompassing inclusive system. In addition to with public safety scores deeming them ‘safe’,
being partial to those with capital, social credit ‘unsafe’, or somewhere in between (Millward
will likely also widen other socioeconomic 2018). Those who are deemed to be a threat
cleavages. Tests and experiments again and are often detained and sent to reeducation
again confirm that data and algorithms are just centres (Foreign Policy 2018). While this is
as biased as society is and inevitably reproduce not the government’s proposed social credit
real life segmentation and inequality (Bodkin system per se—as these types of data are not
2017). Cathy O’Neil, the author of Weapons legally allowed to be collected for public or
of Math Destruction, for instance, warns that market information (Daum 2018b)—the logics
we need algorithmic audits (O’Neil 2016). underpinning this type of coercive surveillance
After all, algorithms are not some naturally infrastructure and the dreams a nationwide
occurring phenomena, but are the reflections citizen rating system are the same.
of the people (and societies) that create them. These developments represent a new
For this reason, the rule of algorithms must reality that, while shocking initially, has
not be mistaken as an upgraded, more rational, become a banal part of everyday existence in
and hyper-scientific rule of law 2.0. This is a few short years. It is becoming increasingly
particularly true in China, where the concept of clear that Xinjiang is a testing ground for
the rule of law has been increasingly developed technologies and techniques that will be rolled
and theorised by the Party-state to justify its out nationwide—and even beyond—in the near
attempts to consolidate control over society future. For instance, over the spring festival
(Rosenzweig, Smith, and Treveskes 2017). period railway police in Henan used glasses
In recent years, China has already been augmented with facial recognition software
providing glimpses of the repressive connected to a centralised database to identify
possibilities of algorithmic rule. In particular, suspected criminals (Wade 2018). China’s
recent reports about the construction of a massive surveillance market is also a global
sophisticated high-tech surveillance state in affair, with companies from around the world
the Xinjiang Autonomous Region anticipate lining up to develop products for both the
a near future where a digital social credit Chinese state and private businesses operating
system sits at the core of a coercive security in the country (Strumpf and Fan 2017). This
apparatus that is inherently biased against points to the fact that China is not developing its
certain segments of society—producing surveillance capabilities in isolation but is at the
dramatically inequitable and ultimately violent forefront of a global push towards increasingly
results (Human Rights Watch 2018). In an op- centralised and interconnected surveillance
ed for the New York Times, James A. Millward apparatuses. Rating systems like the proposed
describes the extent of the surveillance social credit system will inevitably sit at the
146 MADE IN CHINA YEARBOOK 2018ON A CHINESE SCREEN
centre of surveillance regimes, providing the of a digital future where we are all locked in
basis for how individuals and organisations are a continuous and banal system of monitoring,
monitored and assessed, and what they are able accounting, categorising, and tracking—which
to do (and not do) within society. has potential far reaching consequences for
those who challenge the hegemony in any
way, or even just for those who do not have
Our Dark Digital Futures the resources or capacity to participate in the
socioeconomic system on the terms mandated.
Big-data driven social benchmarking sparks
China’s proposed social credit system and entrepreneurs’ and politicians’ imaginations
the ongoing construction of a surveillance about the opportunities lying ahead. And even
state in Xinjiang represent the vanguard of though not all visions will be economically
more efficient means of socioeconomic control or politically viable at any place in the world,
that are being taken up around the globe. They the general trend appears to be global and
are dark outgrowths of the digital revolution’s irreversible. Social credit and the dreams of
supposed ‘liberation technologies’— financialised governance are not Chinese or
underpinned by our very human compulsions authoritarian particularities, but are, perhaps,
for transparency, security, and fairness. Credit our ‘shared destiny’ (gongtong mingyun)—
systems are, of course, not new, nor are they to use a term employed by Xi Jinping when
Chinese in origin. Most industrialised nations talking about the Chinese vision for the future
have been relying on credit ratings for a long of humanity (Barmé, Jaivin, and Goldkorn
time in order to quantify the financial risk of 2015). This, however, does not make it less, but
countries, firms, and individuals (Yu et al. rather more worrisome. The logical conclusion
2015). Indeed, some of the most disturbing of society-wide financialisation is the blurring
aspects of Chinese social credit, such as of the border between political and commercial
its integration into social media, are not realms, and the sharpening of the repressive
uniquely or originally Chinese. In the United tools wielded by the rich and powerful. In
States, Affirm, a San Francisco-based lender China this scenario appears to be inevitable.
headed by PayPal cofounder Max Levchin, To quote Lucy Peng, the chief executive of Ant
has been experimenting with social media financial, ‘[Sesame Credit] will ensure that the
data to evaluate the credit risk of car buyers bad people in society don’t have a place to go,
since 2013. And Lenddo, a Hong Kong-based while good people can move freely and without
company, took an even bolder approach and obstruction’ (Hvistendahl 2017). ■
informed debtors’ friends on Facebook when
they did not pay instalments in time. Even the
Orwellian nightmare unfolding in Xinjiang has
its parallels elsewhere, such as with the recent
revelations that in the United States, the New
Orleans Police Department and Immigration
and Customs Enforcement have been
working with Peter Thiel’s company Palantir
Technologies (which also has connections with
the CIA and the Pentagon) to experiment with
‘predictive policing’ based on data collected
from police databases, social media, and
elsewhere (Fang 2018; Winston 2018). Taken
together, these developments reveal a vision
MADE IN CHINA YEARBOOK 2018 147This text is taken from Dog Days: A Year of Chinese Labour, Civil Society, and Rights,
Made in China Yearbook 2018, edited by Ivan Franceschini and Nicholas Loubere,
published 2019 by ANU Press, The Australian National University, Canberra, Australia.
doi.org/10.22459/MIC.04.2019.22You can also read
