Manipulation and transparency

Fleur JongepierAssistant professor of ethics

f.jongepier@ftr.ru.nlThursday 10th May 2019, iHub lecture series

Cameron Marlow, former head of Facebook’s data science team: “For the first time, we have a microscope that not only lets us examine social behaviour at a very fine level that we’ve never been able to see before, but allows us to run experiments that millions of users are exposed to.” https://www.theguardian.com/technology/2017/sep/19/facebooks-war-on-free-will

“It was one big unethicalexperiment. You’re playingwith the psychology of an

entire country”

Christopher Wylie, Cambridge Analytica

whistleblowerhttps://www.theguardian.com/news/2018/

mar/17/data-war-whistleblower-christopher-wylie-faceook-nix-bannon-

trump

“Facebook was hijacked, repurposed to become a

theatre of war: (..) it became a launchpad for what seems to be an extraordinary attack on the US’s democratic process.”

In recent years some of the smartest people in the world have worked on hacking the human brain in order to make you click on ads and sell you stuff. Now these methods are being used to sell you politicians and ideologies, too.

(…)

How does liberal democracy function in an era when governments and corporations can hack humans? What’s left of the beliefs that “the voter knows best” and “the customer is always right”? How do you live when you realise that you are a hackable animal, that your heart might be a government agent, that your amygdala might be working for Putin, and that the next thought that emerges in your mind might well be the result of some algorithm that knows you better than you know yourself?

the very concepts of “individual” and “freedom” no longer make much sense.

We need to find a balance between taking seriously the idea of online manipulation, without ending up with naïve determinism.

Worries about online manipulation

• Targeted advertisements• Behavioural targeting for political purposes• Recommender systems (Netflix, Amazon), incl.

- news recommendation systems- Home Assistants e.g. Alexa- apps using geolocation to recommend services/products

• Social credit systems (potentially)

Susser et al.

‘Online Manipulation: Hidden Influences in a Digital World’ (2019) by Daniel Susser, Beate Roessler, Helen Nissenbaum

Three cases:1. Leaked internal Facebook document describing how advertisers could use

platform to target teenagers at moments when they feel ‘stressed’, ‘defeated’, ‘overwhelmed’, ‘anxious’, ‘nervous’, ‘stupid’, ‘silly’, ‘useless’, and a ‘failure’,”• Cf. O’Neill’s discussion of for-profit schools targeting vulnerable students

2. Algorithmically nudged labor (Uber’s use of gamification, heat maps, auto-queuing, etc.)

3. Psychographic profiling and election influence (Cambridge Analytica)

The landscape of digital influence

Persuasion

Coercion

Nudging

Manipulation

Akrasia aggregation

Convincing with compelling reasons

Removing acceptable options

Changing choice architecture

Exploiting willpower vulnerabilities

Steering internal decision-making

process

Coercion

To make offers people can’t refuse; to eliminate acceptable options

• Raz (1986): “The state has to ensure that citizens have access to an adequate range of options”

Are we coerced online? Are acceptable options eliminated?• Accepting cookies or being denied access to website (Volkskrant debate)

https://www.volkskrant.nl/columns-opinie/de-volkskrant-moet-betere-privacy-bieden-vindt-deze-hoogleraar~b71fb1f6/

• Do we have a right to e.g. seeing non-personalized options in online stores? Why? What makes the (absence of) options unacceptable?

Nudging

“A form of choice architecture that changes the behaviour of people in a predictable way without forbidding any other options or changing their economic incentives” (Thaler and Sunstein 2008, 6).

• Nys & Engelen (2017) add: “nudging makes use of psychological mechanisms, cognitive biases and heuristics that cause people to make decisions that often go against their own interests”

When is nudging acceptable according to T&S? (1) must be transparent (it should be possible to ‘see’ the nudge), (2) Easy opting outing, (3) should improve welfare

When is a nudge manipulative?

• Sunstein (2016): if it does not “sufficiently engage or appeal to people’s capacity for reflection and deliberation”

• Susser et al. (2019): if the influence is hidden

Hypernudging (Yeung 2016)

The “hyper personalization of a user’s digital choice environment”

Hypernudges steer or recommend decision-making processes yet don’t “forbid” any options, hence seem to count as nudges (Lanzing 2018)

E.g. search engines:

• Yeung: “Although theoretically free to review all the potentially relevant pages (from the hundreds of thousands ranked), in practice each individual searcher is likely to visit only those on the first page or two (Pasquale 2006). Hence the user’s click through behavior is subject to the ‘priming’ effect, brought about by the algorithmic configuration of her informational choice architecture seeking to ‘nudge’ her click through behavior in directions favoured by the choice architect.”

Manipulation

Joseph Raz: “manipulation, unlike coercion, does not interfere with a person’s options. Instead it perverts the way that person reaches decisions, forms preference, or adopts goals.” (1988, 377)

Susser et al. (2019) follow Allen Wood (2014): to manipulate someone is to “undermine or disrupt the ways of choosing that they themselves would critically endorse if they considered the matter in a way that is lucid and free of error”, it is to “deprive them of authorship over their actions”

• Footnote: the autonomous person is not an uninfluenced person; “autonomy is not the absence of influence but the presence of self-government”

Manipulation

To be worried about online manipulation is to be worried about that which it undermines.

• Standard reply: it undermines “autonomy”. - Those who don’t like talking about “autonomy” need to come up with a

different story of why online manipulation is worrisome.- The other option is to take the autonomy route, but to avoid intellectualized

conceptions thereof.

The ambiguities of ‘autonomy’

Gerald Dworkin:

• "autonomy" is used in an exceedingly broad fashion. It is used sometimes as an equivalent of liberty (positive or negative in Berlin’s terminology), sometimes as equivalent to self-rule or sovereignty, sometimes as identical with freedom of the will. It is equated with dignity, integrity, individuality, independence, responsibility, and self-knowledge. It is identified with qualities of self-assertion, with critical reflection, with freedom from obligation, with absence of external causation, with knowledge of one's own interests. (1988, 6)

• It’s a mess.

Autonomy and rational deliberation

Traditionally, ‘autonomy’ is used to refer to internal capacities – rational deliberation –and ‘freedom’ is used to refer to various external conditions, e.g. having options, non-interference by others, not subject to arbitrary power, etc.

• Low key definition of ‘rational deliberation’: being capable of formulating and setting one’s own goals, or: formulating and pursuing one’s conception of the good- Individualist? Rationalist? Atomist?

• Denying the low key definition means denying we have anything like plans in our life and that we can decide for ourselves to read a novel, to buy a new pair of sneakers, to go on a hiking trip, to vote for X not Y, to have children, etc.

• The ability to set one’s own goals is an ‘internal’ capacity, however, the ability to set oneself goals is not wholly internal, because various external conditions can negatively impact one’s capacity to set goals, namely: manipulation.

Comparing the low key definition

1. Autonomy as conscious reflection (e.g. ‘system 2’ thinking)

2. Autonomy as ‘taking control’- E.g. self-tracking community (Sharon 2017)- Controlling what information about us is available to others (Roessler 2017)

3. Autonomy as goal-setting and realization

4. Autonomy as authenticity (acting in accordance with one’s true desires/preferences)

• Autonomy1/2 can reduce autonomy3/4- Taking control over one’s privacy (reading all the terms & conditions), deleting cookies,

installing & tinkering with hardware/software can make it harder to live your own life

• Not: “autonomy is not of ultimate value” but: which autonomy is enhanced & which reduced?

Defining manipulationTo get someone to do something that is in the manipulator’s interest by exploiting the person’s vulnerabilities, and without the person’s knowledge, and were she to know about it she would not have consented.

1. To get someone to do something (intention condition)2. That is in the manipulator’s interest (interest condition)

3. By exploiting the person’s vulnerabilities (vulnerability condition)4. Without person knowing about 1-3 (hiddenness condition)

5. And were she to know she would not have consented (consent condition)

(No ‘authenticity trouble’; still a violation of autonomy.)

Manipulation paradox

“It is only because the target of influence is unaware of or doesn’t understand how they are being influenced that the manipulator’s intervention can infiltrate and disrupt the target’s decision-making process, steering them without force” (Susser et al., 17)

(Apparent) paradox: post-Cambridge Analytica, it seems no longer true that people don’t knowonline manipulation exists, and that they might be subject to it. If manipulation requires a lack of knowledge, then they aren’t manipulated. And yet it seems they are.

• Many are outraged but not particularly surprised when new scandals come to light.

Objection: To assume that people know they are being manipulated gives them too much credit (and big tech corporations too little). Most people really don’t know much at all about the ways in which their choices are influenced.

Any evidence?

• Zarsky (2003): “In ongoing surveys conducted by Alan Westin and others, it is apparent that there has been a growing public interest in personal information. According to Westin, such concerns focus on issues of intrusion (unwanted mail and telemarketing), manipulation (profiling that allows “hidden persuader” marketing), and discrimination.”

• Turow et al.: “84% accept that sites change the ads that people see based on what they are reading on their sites; 84% believe that sites buy personal information about “you” from database companies; and 75% agree that sites change the products “people” see based on the personal information that the sites have bought from database companies. These responses parallel our earlier-noted finding that 80% of the respondents know “Companies today have the ability to follow my activity across many sites on the web.”

• Disclaimer: surveys on manipulation <> privacy

How ignorant are we?

“most users are not aware of hypernudges because they are unobtrusively integrated in most of our online informational environments. Furthermore, they are also not aware that the choice architects behind hypernudges are corporations with economic incentives. Google, Facebook, or FitBit may deliberately steer users in a certain direction without their knowledge of the underlying intentions.” (Lanzing 2018)

At the same time…

There’s a vast amount of literature on people’s ignorance about privacy policies; that people accept terms and conditions without reading them, and would not properly understand them if they did (e.g. Marreiros 2018), as well as lack of understanding about e.g. cookies and what deleting them brings (McDonald & Cranor 2010)

• Smit et al. (2014):- “The results of the online survey showed that the knowledge of [online

behavioural advertising] outperformed that of cookies, as well as that knowledge was still insufficient to understand new advertising techniques. This finding is especially surprising considering the nature of our sample of very regular Internet users. We expect that the knowledge in the general population is even lower, which raises the question of the effectiveness of informed consent legislation because a prerequisite for informed consent is having sufficient knowledge.”

Understanding online behavioural advertising: User knowledge, privacy concerns and online coping behaviourin Europe, Edith G. Smit, Guda Van Noort, Hilde A.M. Voorveld

Reply to the objection

• Heterogeneity in knowledge and understanding: there will be people who have no idea. But most have ‘inklings’; many have more than inklings, and some (e.g. experts) know full well.- Sunstein (2016, 109): “consumers will have diverse understandings of, and reactions

to, statements and actions that plausibly fall within the category of manipulation. Some consumers will see right through it; others will be completely fooled. Empirical testing of representative populations could provide highly informative here.”

• Types and degrees of knowledge: (not) having knowledge of (some) aspects

So, the question is thus not whether the manipulation paradox is real, but just how widespread it is and to whom it applies.

Responding to the manipulation paradox

Two options:

1. Revising commonsense beliefs on the basis of philosophical definition: People aren’t manipulated when subject to e.g. online behavioral targeting

2. Revising philosophical definition on the basis of commonsense beliefs:People are manipulated thus manipulation doesn’t require hiddenness

Costs of option 1: intuitions should be revisable on the basis of theoretical distinctions, but this one seems too revisionist/hard to swallow. So how about option 2?

Against hiddenness as a necessary condition

Sunstein: “some acts can be both manipulative and fully revealed to those who are being manipulated” (104),

• E.g. relative risk information (‘your chance of a stroke is doubled if you don’t take these pills’), or explicit warnings: “This movie contains subliminal advertising.” Cf.: “This advertisement was brought to you through behavioral targeting”

But,

• Not evident these are cases of manipulation at all (‘The government manipulated me into not smoking!’)

• Isn’t there still something importantly nontransparent about online manipulation even for those who “know about Cambridge Analytica” and “commercial incentives”?

A third way

People sort of know.

• ‘Aspect knowledge’: (not) knowing aspects of a proposition

• ‘Contextual knowledge’: (not) knowing in certain contexts/frames of mind

Aspect knowledge

Full knowledge that one is being manipulated would involve having detailed knowledge of who, what, why and how they are being manipulated:

• Who manipulates- E.g. Cambridge Analytica/Facebook/‘The Russians’

• The action type (what manipulator intends to bring about)- E.g. to get someone not to vote for Hilary

• The reasons (why the manipulator intends to bring about X)- Because of a desire for Trump to win

• The means/strategy (how the manipulee tries to bring about X)- Knowledge about data collection, data storage, data analysis, targeted advertising

How much of these aspects does one need to know for it not to be true one is manipulated?

Aspect knowledge

• The 60% in the Smit et al. study may perhaps have partial but sufficient knowledge of the “who, what, why and how” and so it would follow they aren’t manipulated. (And so Harari’s claims should be taken with a grain of salt.)

• The asymmetry makes things more worrisome rather than less, however, as it illustrates the problem of epistemic inequality- Smit et al. (2014): “The users that are most concerned [about misuse of their

personal data] consist of older people, females, and people with a low level of education and a low family income. This group had even less knowledge about [online behavioral targeting] and cookies than the other groups.”

Knowing about the means

Uber’s sending push notifications to keep drivers on the road: “Are you sure you want to go offline? Demand is very high in your area. Make more money, don’t stop now!” • Susser et al.:

- Manipulation? “as it is neither hidden nor significantly targeted we would ultimately not judge it to be so. As a nudge, it is designed to exploit our desire to accomplish goals (even if they are functionally meaningless), but it is not a manipulative practice”

- If we “tweak” the practice it might become manipulative, e.g. if notifications were timed when drivers were despite and earnings goals were “indexed to bills coming due” and drivers wouldn’t know this strategy behind it

- So we know who + what + why, but not the precise “how”, and that’s why we’re manipulated

• Must companies inform people about the how; their concrete strategies?

How transparent should companies be?

‘Online advertising (onterecht) onder vuur’. Guest: Jeroen Verkroost, board member Interactive Advertising Bureau

• I1: “It has to be transparent how one is being influenced (…) otherwise we’re only puppets.”

• I2: “I’m not sure if companies are obliged, if they do this, to make explicit how they do it, because then you would reduce its effect. It would be foolish if Facebook would have to publish how they organize it all”

• I1: “But awareness of being manipulated would be good” (…)

• I2: “But should a company that advertises make sure that you’re aware? I think that would be strange, actually.”

• JV “In fact, it’s the [geheim van de smit]. What we sometimes encounter at IAT, when we notice things that work really well, that we ask a marketeer or publisher “would you like to come on stage and tell others about how you got this fantastic result?” and they often say, “well, I’d rather not”.

Informing people about the ‘how’?

If companies aren’t going to be more transparent, we can still educate people about the strategies regarding online manipulation in other ways.But: tradeoff ‘autonomy’ and ‘autonomy’

• Encouraging people to engage in critical thinking and informing themselves about online manipulation makes them more autonomous (1/2) but can also reduce their autonomy (2/3) because people want to pursue other goals than read terms and conditions on Sunday morning.

• Related worry: emphasizing people have full knowledge typically shifts the responsibility balance (‘you should have known’)

• Being more transparent as intrinsically good <> repairing damage

Contextual knowledge

To (fail to) know in a particular context or frame of mind.

• James Williams: “Epistemic distraction is the diminishment of underlying capacities that enable a person to define or pursue their goals” (68)

• E.g. through fake news, “e-mail apnea” (a phenomenon that occurs when a person opens their mail to find many unread messages inducing a fight-or-flight response), social media stress, and “cyberchondria”

Epistemic distraction is probably much more pervasive (e.g. infinite scroll, ‘red’ notifications, push notifications), and also the very design of certain technologies (e.g. ‘flow’) can make one temporarily forget/block knowledge of online manipulation.

• If that’s true, anyone can be manipulated online, despite “knowing about Cambridge Analytica”

The last slide: the other approach

Manipulation is a success term: to manipulate someone involves succeeding in perverting her decision-making process. • Hence, the focus is understandably on the effects on the manipulated individuals, and

therefore, how manipulation affects ‘autonomy’ • Alternative: to focus on the manipulator. Manipulation is wrong because those

engaging in manipulation have an immoral attitude in that having manipulative intentions involves treating persons only ‘as things’

• The “bad” news... - “So act that you use the humanity, whether in your own person or in the

person of any other, always at the same time as an end, never merely as a means.” (G 4:429)

The real last slide: so what do I think the big research Qs are for iHub?• On what conception of ‘autonomy’ does it make sense to think people are manipulated

online?

• Who is manipulated and who isn’t, and how does people’s different epistemic capacities affect this? Which new ethical challenges are introduced by this knowledge asymmetry?

• Which ‘epistemic goods’ would people most benefit from? How far does e.g. the “right to explanation” in the GDPR go in this regard?

• At what point does knowing more become autonomy-undermining?

• Answering these Qs involves collaborating with- Social sciences, to know what people (don’t) know- Computer science, to know what people could come to know about the ‘how-question’- Law, to know what people have a right to knowing & how manipulation as a theme can be

tackled in law (Zarsky 2019)

thanks for your attention