:;;;;;tttt%t%%%%%S%SSSSSSSSXX@@XXXXXXXXSXSSXSXXXSXSXXXSSXXXXXXXXXXXXXXXXSXSXSSSSS%S%%%%%tttt;;:;: :;;;;ttttt%%%%%%SS%SSSSSSXXXX@@@@XXXXSXXXS .8SX8XS@. ;%SSXXSXXXXXXXXSXXXXXXSXXSSSSS%S%%%%%ttt t t;;tt S%t%%%%SSSSSSSSSXXXXXXX@88@@X@XXS88X8t8X8X888888@88t8:S%@XXXXXX@XXXXXXXSXXSXSSSSS%%St%S;%SSSSS%tt XS%t%%S%SSSSSXSSXXSXXXS@X@@888St8X8S@@SX888X@88X8888@888X8 @@X@@XX@X@XX@XXXXXSSSSSSS%%%tttS%ttS;:t%%%%%S%SSSSSXXXXXXXX@XX@@@@X88SX.8888X8S88XX888888@88X88888888t@XX@@X@X@@X@XS@XXXXSSSSSSS%%t;tX.:;t%%S%%SSSXXXXSXXX@@@@X@S8@8@8S888888X888X888888XS888X8888XX88X@88%@@@@@S8@@X@@X@XXXXXSSSS%SStt;:;;S:%%%SSSSXXXSXX@@@S8@S8@88@8S8;%88888@888888S8@8X@XS@S88@8888888X888:@S888@@@@@@X@S@XS@XXSSSS%%S;tS:;;%SSSSSXXXX@@@@@@8@@8@@8@S8@88@@88XS8XX8@88888888X888@8888@X88888888;8@@@8S8@@@S8@@@@XS@XXSSS%%%tttt;SSSXXX@XS8@@@S88@888S8888X88888888@X88X@@8@@88@8888888@88S88S888X888.88S8888S88@8@8@@@@@X@XXSS%St%S;SXXS@X@@8@@S88888S88888888@888@8X8X88@88S88@@88@88@@88888888S88@8S888X88888888888S8888@@@@SXXSS%%tttXXX@@@@@@8888888888888S8888X@S8S888@@@88888888@@88X8@8X8888X888SS%@@@:88888888888888S88S8@@XXSSS%%%%X@@@@S88@S888888888888888XS8:88888X8;8888888888888888@8888XX88X888@888%8888888S88888888888@@S@SSSS%%X@@@888888888888888888S8@S8888X88888@88X88@@88888X@88S88@88X8888SX88S8t888S888888888888888S8@XXXSS%%@@@8888888888888888888888@8@888S8X88X@88S%SS8888888@8SX@S8888888S88S88@.8888888888888888S888@@XXSSS%@888888888888888888888@88888888;8%88888@888888@X888888@8S88X888888888S8;.888888888SS8S88@8888S8XSXSS88888888888888;88;88Xt88XX888X88@8 8:888X88X@8S8888XS88888@S88@8@888S8%8S@8888888t888S88888888@@@SSS8888888888;8;888888;888X8X%888888888888XX8X88888@8888888X8S888888888@XX88S8888tX%8888888888S888@X@XS88888888;88888:8:88888S88888X 8@SSX@@X8 @@@8X888;:88%88@8.8.;88888S8S;88%888tXS8888888888888888X8XXX888%88:888:8:8888.8t@888888S8X@8.88X888888 SS SX8@8888 X@%88X888t88X88:8888S8888t88888888888888@@X8X88t888:8888:8.88;888S888X8X8@XSS88888X .8 X@8888%@tX888SX88XX888@888:S8%888%8888;S;8888888888X@@88S88t888:8.88:S.88;8X.88888X88S8S8@8888@S %S@8X8X :S8@@8XXXX8 8X88:S8S@8888t888;St8S88tS;8888S888@88X8t8;8:888.8;8;8:8t8;:8S88. %X88S%8 S888S SX@88888S8X@XX @XXXS;8@t8S8@XX8t888;S%X888888t88888X88@@8@8%88t88.8.88:8t88:8;S8;8%88S@X8 8.X88@SS X@88888@S@8S@XX%SSS @S88@8S888X88t88t888t8t88888888888888@X8;8t8:8:8:8;8;;8;8%8%8S8@% S % SX@@ S SS@@88SSS X@ X%@8 XS8888.8;8888%888t88t88888;88;888888888@X8t8;8;88:8:8:88.8;8;8XX.@% % % SX@%%X XS8888X@XSXS XX8S@SX8X88 .888@X8t8;St888;88;S;888;888888@@XS8;8;8.8;8;8;8.8:8:8;XX8 8% S S S%% XX%XS@@@8@@@XSS X X @%X8 X 888@88S88%8S8;88;88t88;S%888888@@@SS8;8;8:8.8:8.8:8:8.8tXX8 ;; 888S8S X@ XS88X@88@ S S%@SX888 8@@@XX8t88t888:888;88t8%88;8888@XX8t%8:8:8:8.8.8.8.8.8:SSSXt@S 88 8XSX S SS88 88XS S 8 X XX8S8;S@@@@X8t8;8;8;88:88t88888888888@XXS8t%8:8.8.8.8.8.8.8.8;SSSS8 8X8 888 S S8S SX888@X@SXSS S SX8S 8S;X@XX@X8t88;888:88:88;8;8;888888XXXS%8;t%t8.8.8 8 8 8 8 8;SSX@88S 88 8X S%SX%8@X8XX8S8%XX@%8SXSX 888SXX@X8t8tt88:8:8;88;888888888888XXSX%%S8tttttttttttttt;8 %%S%S@8 S 8SX SSX%X8X88 SX@@8@8S@ SSXSS8%@XXXXSS8;88.8;8888.888.88:S;88S88XSXSSS%t8;8;8;8;8;8;8;t;8:S%%%SS8 888S SX % SX@ X@8@S% S%@S@SS88@XXXX8t%8:8;88 8 88:888:S%8888888XSSSSSS%tttttt;t;t;t;8:t;8:%%%%SS8%8 8@X8X@ 8 8 : 8@8tXSS SS@8SS8 88:8SS%8t8;8;8.888.88.888%88888888SSSSSS%S%S%%%8:%%%%%t;%8.;t%%%t%SX8S S S%tS8@88888SX8S XX %8XS@SS 8S %8SS%%t8:8:8.8.88:X;888888S88X8SSSS%S%%%%%%t;%%%t%%t%;;%t%ttt%ttS@88S8S . S X X SX@ 8X8XXX8@%8SX%;;8%8;8:8;88.S.88%888888888888S%S%S%%%%%%%t%tt%tt%t%tt%tttttt%tt%@88 88 8X X S%@S XXX8S8XS@ X 88t%% ;X888;8.8.8;8;8888888S888888S%%%%%%%%%t%tt%tttttttttttttttttttt%SX%8@ @ SSX @S8@X8S8XS8 S%X 8X%X%t%% SS@ 888:8t88;888888888X88%%%%%%%%ttttttttttttttttttttttttttttttS88 SSX % SS8X@@8@S S S%8.888X:;S;%%.% XS %88888:888888888@%%%t%ttt%ttttttttttttttttttt;t;ttt;tttt%SS88XSS XX@S8 S 8S88888%%X%SSSSSXS ;tSSXX%8S88888888%tt%tt%tttttttttt;tttttttttt;;;;;tt;t;t;ttS8 XSX S SSX S 88S 88888888S.X;X@XX@8XSS%%tt%S S888S88ttttttttttttt;t;tt;t;t;t;t;ttt;t;;t;t;tttttX8%X 8S 888S 888@8%@88t :8@8@tX8;8@@@@XX %.tSS;%%tttttttttt;t;t;t;;t;t;t;t;t;;;;;;;;t;;t;;t;X;888 8 8 ..88 8 888@@88 88t8888X8 %888S:@88888@@S..%ttttt;tt;t;t;;;;;;;;;;;;;;;;t;;;;t;;;;;;;;;S8;8%8 888 8 8 88S88t8%S8888888888888888@@8X@8@88t %XXX@@t;;t;t;t;;;;;;t;;;;;;;;;;;;;;;;;;;;;;;;;;t%X@S 8S88 8 8 88888;XX8t8X8t 88888 888888@S@888%8 X88@XS;;t;;;;;;t;;;;;;;;;;;;;;;;;;;;;;;;;;;;;tS8@88:S88X 8%88 8 888SS%8%@8888 88888 X 888 888 8888S88% S8;;;;;t;;;;;;;;;:;;;;:;;;:;;;:;;;;:;;;;t@8X8t8X@@S 88S88 88X%SSt%@@8888 88888888888 .8888888888%SS8@;;;;;;;;;;;;:;:;:;:;;::;;:;:;:;:;:;;;S8888X 8S%XSX. 8%8888St%S8X88888:88888.88888888888888888888888;;;;;;:;;:;:;:;:;:::;:;::;:;:;:;:;;X888@8@8X@8@X.8@888888%;t8t8%XX@88888888.888SXt8888888@8:88888888;;:;:;;:;;:;:;:;:;;:;:;:;:;:;:;:;:%8@8888X@88;@888%XX888S%%tt%%SXXX@X888888888 8@.@88@8@88t888888:8@;:;;;:;:;:;:;:;:;::;:;:;:;::;::;:;8@8S@S8@XX8%@88@8%%X8@%SXS8X@@XXXS8:8888888888@@8@88888888@88 X 88;;::;:;::;:;::;::;:;::;:;:;:;:;:;;@@@S8t88XX@X8888@8;88t%%8;X@8888888@888 t 8888888@88888888@@888@8@t:;:;:;:;:;:;::;:;::;:;::;:;:;:;:t888SSXXX@XXS8888@@X;t%%%8%Xt.@@@@X@@@88S;@8@8%S8@@8888:8.8X@@@X@88t;:;:;:;:;::;:;:;:;:;:;:;::::::::t88@X@SSXS@%88888@X8@XXS%8%X888@88.8X@@@@@@@@ 888@888888t@S8X 8888S;;::;:;::;:;:::::::::::::;::;:;:t888@@SSXX@8@888@X@88:%SXtXX@@XX8888@SX@X@@@@8888@8@888@XXXS.8 888;St;::;:;:::::;:;::;::;::;::::::::t8@@@X888XX@XX@S8;8 88X%8:SSXXXX888@XXX@@@@@@@88@@@@XXSXSXSS@8888 8X%;;::::;:;::::::::;::::::;:;::::%8@SXX8XXX8XXSSS8@888XSSX8XSSSXXXXXXXSXX@@X8 S88@@@@XXXXSSSXX888888XSt;::::::::;::;:;::::;:;::::::::t88%S@XXX8XXSSSS@@@X@X%%@ XSXXXXS88@88@XXX@@888@@@XXXXXSSSXX@8888 ;XX%;::;::;::::::::::;::::::;:;:::t88Xt@X8@XXSS8tS%X@8 88X8 @SXSX@88888XS888X@X88X8@X@XS@8XSSS888:8 8XXSt;::::::::::::::::::::::::::::;88SXSXX8S@SS88 88@8X;SSX8XXSXSXXX@X@S88;XXX@88 8XX@S888%@X@88888 8XSS%;::::::::::::::::::::::::::::;88SXSXX@8@%S8@88@XS%;t%X8S@SSXS@88@X88%SXXXX@8XSXXX@@888SXX8@88@8

DEPARTMENT OF SCIENCE, TECHNOLOGY, ENGINEERING AND PUBLIC POLICY

Lilian Edwards [@lilianedwards]

University of Strathclyde

Michael Veale [@mikarv]

University College London

Oxford OBOR Summit University of Oxford, 14th September 2017

%.:t%%X: :S8XXXSXS%8t%.;.8%8.8888.%;;:88@88888:888888t@%S@X8XXS:S;.tS:% 88 8 :;X@S;XX .:: t X ;.8;8%;::SSXt8888%8@XX%8;@8@8X.X8X8@888XtS8X888888X8888t8:88%X8X8888SS8:%ttX8 %S@8 8.;% ;88X SX%@%@t8t:; %;S888;8888@8@@XXXX8888888 88;8888:8.88888X8@8t8888888 t8%XX8@8@X888:t8: t8 8t8 tXS X X@;% 88:8 %:XS8@%S8;888888@X@@%X:X888S8.88.8.:8X.X888 8888S88X tS 88t88S8t%88888Xtt .8S8S@ 8 :88@.S:@ :t8@%@8XtSXS8SX888888%8@8XX8;888S8.8t8%88:t888@8:X88 88888888888 8;88t8t8888;@XXt.t@;888:;%t:8S.@SS;%;S@:8@%t%tt8S8888888X@@XXSS8.S888@8888@8888 8S ;:8%88S8;88S88888%8888 8.:8.88@S%@8t;:8X.8%;@8t%8 88;:tX88S%..%X888t888%8X8S%888888@@XXSXXX8SX88888X:t8 t8888%888888%8 888888888:8888.S8S;.88:8X S%: :@.t%@8@;St%.t%8SS88 @8tX@S@8%888@XXXXX8:8 8S88888%8 S88t8:88S8%88:.8888@ 888@8Xt%88.S@X8;;88 :8X X% .8S.%XS 8;X 88t%88SXXS%8S888@8%8 8 88S88S8@88S8%888;XX@S8@888888;X%88;@88@88888..X:%XXXt 8.88:S8;8 ttt:S@8tX%8S8888.8S8XXX@%88@X@ 888 8888XX@888X8%8@@XX@@@8888 .88:..8 ;88888:888@;@S;XS8t88;;;88.@t.@SX.S%@; ;

.88%888888888tt88;%88 8S8@ S888S@8 88888@@8888;88888@ %S88 ;X.88888 88888SX%S8@:8..88: 8XX8;:@8888@;@S.%:88.S%8%.888888888X8888 8@888888@888.XS8@X888.%X:SSttt88.8;t8888888S8X8@t.tt8 .8:t: .S 8 X88@S.:8%8:88 8888Xt8%SS 8 XS888 8X8X88@8@8888X@t;88X.;:tt%88@8tt88;;;SS8%X%8888 88 ;X%S88;.::8;@ ;8 %%%..8:;88tX@..8:;..8888888@S@X%8 8 8@8XS@888.888@88888888X8888t8S8t.%SSX8;.888X 888SXXX%St :X88 8:.t;:%:8:88 S88.:%;%XS%;tt@S8;8 8888;88 @S;@X%;8888888 8.8:8.888888%88.8@@8.XXt8@88XSS.;X;8t88tX @ 8%.t.%:;888 88:tt8@8888@@%%%8tXSX88%S: 88 S8;888.88@X8888888 88888X @ ;8.8;::88888t; 888t8888 8t@@8tXS.t;.8@ 88888:S@XSSS8@;XS tXS888:88X8;888 8@8@:X8S888888888;%88@88X88888888t@88;t;8S%8888888;88@XS.X888888@ @8888X@XXXX%XXSS@@8XtX88;8. 888t;8S8SS@@SXStt888888888XXS:8.X8888888X8St88888.8888888X 8888;88@88 X88@@XXt:%X8@XSSt;88%888:888@:888888@S.:SS8@.;.;888;888.88XXt88 88:X888@S88888%888X888@ 88888% 88@ 8 SX88888S8888;.:8 S88@SX@.8S88888888..:;;tSt .888888S888.8@t%@:;.8;8888888@888t.8X@88X 888888.:8888 8@@88888888%8X8888X8X@8 8888S:S88888:8@.t@XX88X8888888S8X8;8888888888%8888888X888S88X 8888888..;:8@S8@@888888t8:888888@88 88X888888X@888888S8;88888888tS888X;8888888888888:8@8S888S8@8.X 8 888888.. ..S 888S888@8 888:@:@88SS888@;8:88%88888%88@8X88%888X@888%@t8888@@88888888%XX8@888888 XS%888@8 @8;.:8888%XXS88;8t8X8; @X@ XS888S8@8@88888;888XS 8S8:888:88@88@%t8888@@@88888S8888888;;88:8@@ 888@8888:88S8%S888 @ X X S 888X88888S8X@@88@@@@@X88X88:8@888X8%888@888888X8XSS;@@88XX88@.t888X@8@ 88@8 88@8888%.88888S888888888@ XXS88888S8@888X@X@XS88%%8%888XXX8 @8;8X888@8X XS 8@88 S8888888@88X 88@8888888SS88888 8 8%X@8@8@8:X88 8888@ 8S8@88888888XS @@888888:8;88S8X@88%88X8888@SX8 X@8888@888X 8S888 8X8@%888 8888X8%888X:@;S888 8888S8X88 88S8;88.888@X888 S 8S8@ 8 XS88SSS88@888 SX8X88 S%8888 8888X8@@%%SS@ 8888888888t::8%8888@S888888 8 8 88@X@X88@ 88X8888888888@88 8888888888S8888SXX8@88@88t 888888@8@ 88X88X88888 @8;X88;8tXXSSS88888@8888 8@8X888S888888@88X8 8 8S8XX88@88888@8X888888t88888 S8888S@ 8 8 @8S888@8888@ 88 X%8S8@@X8@ X88888 8:8%8 8 X8:88@888888@@@@8@8%8X88888888:8888@S88t88888@X88@X8X888888SSS8@888888@888888888X888@ 888@S88X8888@ @8SX88888S888.X8@8X@X@8X888S88S8@@8@8%8SX:888888;8:8888.S888S888888888 8@88888@8@ 8X8@@@88XX8X8t.888X88;888888888.X888888S8 8X8X S8@888@S@:@8 %8SS88888@%Xt8;888X 88@88888 S8@88 8 tX8X8 888t8 S8X 8:St8S@S888@t8S888888;;;88S@X8@8S88888X888:%;8S88%8.X;8%@88888SX.88@88@88888@@888SX88X;8%888@X8X t88.8 8888X8@888t;8t@%8.%88t88@8@8SS8 8 S8XXX@@SX8X8t88:88%S888;8888888888;%88@88888@:@8@ X;88.88%..X S8tS88%8888S8X8@@%888888@8;8888@8S 8888S:%888888@88.88:8X8888888888t8t8888%8tS8S%@88888SX.8.8%888:t8XS8S;8888 88S8X@.888.@8@ 8@88.%8XS88S88 ;@@88X88@X8@@;88888888888X8 8.8:S@888@8@X8.888@8X88888X 888888 X888888888;8888%@8888.888888S @S8 ;%88@88@XS88SS888888888888888888:8%8@8888@@8X8.88X8@X:8%88@.88XX8@88888X8tt.888888888S88888@@XX8888@X8X8S888888@%X%8@@8888@X88@888:t8%888888S88888X%88X@@888S8 8;8.888 8t@S88888t88S88888.:888888S8;.8tt8@@8888X8S8X8888@X%S;@8@88888;8t@8888XX8t8888S8@@88@8X@8888888888X88888@S8:88:88X@t8S8.8.8.:..X;:X88X8X8@888S88888888@8888%8X88@888X8@88S8;8888@S8S888 XX@X8X8%8;8888888;88X%X88@8%888.888X.. ;;8@8t88@888@8@8@88888888t888X88X8888X88888888888S888S8888@88;@S8 8SX888@888888:8888:t%88XtX;88 88.8@@X@8888@8@88@8@X888X8St88@88888888XX8888XS8888S8888@888@8XS8:X 8 8XXX@8@888 @X888@X88Xt;;t8:;88888888X88@@8@@@8888X888X;888888888@888S888@888XSX8S8888888888X88%8@8S8S8X8XS@X@S 8@888888 8888;XX8@88X8XX8888@888X8@88SXX8888X8%8;888S8X.88@888X88888X888888S88@SX@8X@X88.S8X@888S8@@8X888. t8t%8%8X8X8XX@SX8@X8X888X888S8;@88888888:8t8XX88X88@S888X8888S;88XS8XS888@SS8@88t888S8S88@@888%88 %8SX8X88S8@8S8X8SX8@88X@8@88XX88XS88888888t@S88SS888SX88@888@88S8@888S8 @88@88888@8888..88 8888:;8:88@8SX8S8XX8S8X@S@@88X8S8X888S8S8888S8XX8888@888888888888888888@888X888S88@S@S8S888@88@:t8;8888888%888tX@X88%@X88S@SXX88S@8888888888888888888@8888%SS;88888X888888X888X88X88S8888@S888@8888:8S8;888 88888888;88S8t8S8X%SS8X@@@888888S@888888S888S88X88888@888888S8S888S8888888X888@S88@S@S88@88@.8S;888S888S88.88.;8:%S8@S8%8888%8X8@88S8888888888888S8888X88S.888X888888S888XX8S88@88X88SS888@8888@88888;8888@8.8;88@S:@:8SX8XS88X88@8 %8X888S@88S@X88888XS8@8888S8@888888X8:8888888XXX88@8XX8S8X8X@X8SX8 :88t88888888%88X8 8S8888@88888888S8X8X8;88888888888888X8888@8XX@@888888888888S888@88@88X8X8S888S8X;88S8888t88%@88%88S@8@88S88@8XX@8X.8X@88888@8888888S88@S888S8@@8@@X8888888888@88@888S888888@8@8SX@S@@88888%88.8X8888t88S888;88888888S8:8@XS88@8:8X88888X88S88X88S8888@8S@8888S8XXX8888S8%X8888@S8S@888@88%8%8S8;@@8888888t88%8@X@8S8X8@888X88S888:X@X8888888S88X8S8X@8@S8X8;88XXS@88S8888888@8X@@XS8S8XX@XX8X8S:X888:88888:888t888%S888X8X8X8X88888S8XX88@8888888SX88XS88S888@X88888888;88@888888888X88@8S@S8S88S88X.%888;88888:888S88@88@88X88@8888S8888888S8S888@888S8888S88X@88X88S888888@8@888X8SX8888X8@X88S8S8S888;8%88888888@88S888888S@@888S8XS@8@S8888888888888888X8X8S8@888@88@88SS8S8S888888S88888@8S88@8X888@888888%8%@88888%8888888@88X8X8888@8X@8@S888888@8888S88@@888S8888

Slave to the algorithm? Why a ‘right to explanation’ is probably

not the remedy you’re looking for.

Overview

Concerns around “black box”, algorithms, increasingly machine learning

algorithms that improve with data, have fuelled calls to “open them up”.

Are there rights to have these systems “explained”?

In the General Data Protection Regulation (GDPR), maybe.

Applies globally from next May if you are processing data of individuals in Europe. Fines of up to 4% of global turnover if non-compliant.

• Decision support systems • Parole • Money laundering • Tax fraud • Policing • Hiring/firing • Event detection

Algorithms we might care about

• Decicion-making systems • Finance • Insurance • Advertising • Pricing

Using machine learning for prediction

Characterising and utilising patterns in data.

age income education will repay loan?

0.8 (Y)

0.7 (Y)

0.2 (N)

0.4 (N)

Anticipating what hasn’t happened yetage income education fraudulent

claim?

0.3 (N)

0.6 (Y)

0.9 (Y)

0.1 (N)

Detecting what we don’t know about yet

What’s all this about a “right to explanation”?

✦ Backdrop: there are claims a “right to explanation” exist; claims it doesn’t*

✦ We claim that in part it might, but! ✦ It wouldn’t trigger in cases that have worried the public most ✦ It doesn’t link to state-of-the-art explanations in computer science ✦ IP problems are less of a problem than people think ✦ But explanations might not work well for those that need them most ✦ Rights-based approaches are often not the remedy you are looking for.

* Goodman & Flaxman 2016 ICML; against, Wachter et al 2017 IDPL.

The GDPR ported, with little change, a corresponding right from the 1995 Data Protection Directive (DPD) art 15 to a new art 22 which provides 

“the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects, concerning him or her, or significantly affects him or her”

* Goodman & Flaxman (2016) https://arxiv.org/abs/1606.08813; ** e.g. Kobsa (2001) (doi:10.1007/3-540-44566-8_52)

GDPR, art 22 Automated individual decision-making, including profiling

Art 22: Right not to be subject to automated decisions

• art 22 GDPR: right not to be subject to an solely automated, significant automated decision.

• Existed since 1995, but considered a “second class” right (Bygrave)

• Where you are subjected to one, safeguards must be provided. Implied that you are told of the decision occuring, but also mention of a right to an explanation. Yet textual mess in GDPR means that safeguards in recital don’t match those in article (Wachter et al 2017 for more).

First-glance limitations of GDPR, art 22 Automated individual decision-making, including profiling

1. Not an explanation remedy: limited to: i. preventing certain processing; ii. inserting a human-in-the-loop; iii. implicitly, a right to be informed when an automated decision is being made.

Furthermore, only applies to 2. decisions made “solely on automated processing” 3. decisions which produce “legal effects […] or significantly affects him or her”

* Mendoza & Bygrave (2017) https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2964855 drawing on Bygrave (2001) (doi:10.1016/S0267-3649(01)00104-2)

..:;;%%SXX@XSS%t;:.. . .. .. . ...::::;;ttttttttt;:: .%.@;XtXt: .............S;:S:tS:%t%tS;t@:.

..:;t%SS8S8S8;%S:;.. . . ..::::;:;ttttt%%%tt;;. :S:8t@St@. .. . . . . .. . .. . ...... ..

..S.StS8S88:8 tt;S .S %. %;8%: 88; :::;%t%tt;:..;Xt8tX%@%S ........... ..X:;%t;t%;;tt:. .;

...:;S;%888 8 8 ..: S S;.%.t;88%888@t%%S@ : :Sttt;: .t8S8t@XS@% ... . . . .. t8t8.X8X 8@ 8%.. ;

.S ::;t8S8%888 :% .S ..:88tS@tttX;t%tXS;@%Xtt;:. :%8X@%XtX@S... ..... .. %S8@8:8%%88tS8;: t

. S S S . . % .S8tXtS;888888%@@SX8:%tt;:%t;S t@t@Xt88..... . ..... .;.:t; ;; :t..;..;% 8S8S 8 S S % tt%XS@@8S8S888888S8St;; ;tS8SSXt@t88.. ..... . . ..... ...... ... ..

..; % .S % S % :S:XSS888888888@88S8:X;;% t%8 %tX%@t@8.... . ....... ;; .. . . ... . .

.. S S % S S .8:X@88S8XX888@8888t.t;. ;S@8 S;@SX%X@.. ..... . . ;t8StS;S%S;%S. t;t;

..:t S %%S S S % .8;8X@8@8@X@88@8@t%@@:;: %S%X8.S;St@tX8. .. . ..... 8.tXt@tX.X%%8: %%tS

..S % X %SX S % % @;@88X8XX88@888888X8;.. %%@8t%S@SX;XX... ... . .. ;%S%%S%S;SSSS;t;%S%:..% S 8 S% S % % S 88@8888S88X88X88X8X%%:. % SX8:;8%SSSt%X .... ..... .. .... . .. ...:.: S SS SS %X% S % 8S8@8X88X88888%8S888 ;%%@88XS@%XX;:% . .... . ... .............. ...::S;S X @ %X SSS @ % S 8X@88S88@X888%@888888 S %X88;;@SXt@; t .. . ..... ... . ...... . ... .::: S S @ S S @ @ @ XS88@888888t88888X@8:8 S%@88:StSSSX; : ..... . ... ...... . ...... ...:::S S S S@ S SS % SX SS%@8@8X88X@888@88@@ %tt%@8X:SSStXX;X;... ..... ... . . .... . . ... .:;;;S X X%@ SX @ SS 8 @ 8tt88@88@S8888S88XS@ t S%S@8X:X;X%XStX; ... . ... ....... . ...... ...:;X tS S S @ % %X @ S8;S:;S8S88@888@888X@8 tt X@88..@tX%@@t@: .. ... ... . . ...... . ... ...;:;;S S SX X X S 8@8XX88%%tS;8X@8XX88@8S8S88.:; XXS@88.:8tSSt@;8: .... .. ....... . . .... ... ..;;S:;S S S@ S %8S8@XSXS%%%%Xtt%S88@888%@S88%;8; tt%%X@8 8X@tXS%@t8:.. ...... . . ....... . .. ... .;:;S %SX 88:;@88XS%SX%S%%Xt%t%S%X8888@@88@S8@t;8 %;tX888St%SSS@XS;8.... . . ...... . . ........ ...;;;;S @ 8%@8X@S@S@%XtXtXtS%SSXt%t8@8888888888X%%8@8X 88%S;@%@%Xt@tX. ....... . . ...... . . . .. ...;;X;tt XS:%@XtStSXStX;ttX;XXStX%tS88888888888@ @;8@@S%8X .@%%XtXSt%: .. . ....... . . ........ .. ..;:;S X %%St%XtX%%XtXSttS%t%%XtXt8888;8%St..@;%8:%8X8SXS8@@X%X%S@tt:.. ... . . ....... . . . ..... .;;;;% 88@t%%ttXt@%%X;StttXS8t%@t@::S8X8888XX888S8t@@X@%X;%888%8S%@t: ... ...... . . ......... . ....;;X;S 8tS%@tSt;%%X%SS;t%t%@t8SSSt%XS888888@@@888 @t@8%Xt@SXSXX%8Xt:.. .... . ....... . . . ..... . .;;;;%8SXt%%%%Stt%%X;SSttXt%@%@t%@%St8888@X@@X@8888;S@8%X%%@St8SS@@:... . .... . . ......... . ......;;X%%S;StXSXttXttttttSS;;S;%8tXXt@tStX8@X@XX@@@@@X:;S8@S8%S8tS8t%8.: .... . ...... . . . ..... . . .;tt:S@S%XtSS@%;%Xtt%%@;;t%S;%8%StS@t8tSXXXXXXXX@8@;ttX88S8X%@t%8t8.: .. ..... . ......... . .......t;tX8tSSt@tXtXS%tX;;;tS;t%%%tS@%@St@::tXSXSXSXXXX@8 8%888XS@SS%X%8.: .... . .... . . . ..... . . ...tt:;%@t%Stt%t;t%%;%ttt%tStX%tSSStXStX%.tSSSXSXSXXX@8 8S@t@@SX%X;8S. .. . .... ......... . ....... ..%;88XSXtSS%XSt@tXttt;;@;;tt%S;;XtStX%S88XSSSSSXSXXX@%;%t%XX@SXtXt8X@t..... ... . . . ..... . . ... .%;S%8;Xtt%X;t%%St%%t;ttt%%XtS;t%X;SXtSt@t@XXXSSXSXXX@8StXtX8@S@t%%X%@... .. ......... . ....... ...%S8;%%SXStStX%XtSS%t;ttSt%ttt%%t%X%%XS;S88@XXXXSSXXX@XS@%8@S8@X8X%t@X8;..... . . . ..... . .. .. ...t.tX%Xtt%X;X;S;%X%tt;ttt%%%%St%%tStX;ttt.888888888@@@X88t%SX%@XXX;t%S;%8 .. ....... . ...... 88X ..t%ttXtS8%%%%%%S%%t%t;tt%%%%tt%ttttS;S%X: 88;888:8.8.8%:XS@XXS%Xt@SttXttS%.... . . .... . .. 8@8; : .@@tXtX%%@%XtS%@ttS%%;;%tt%%tt%%t;tt%t%%t%t88 .: % @@%XSt%X%XX;t%t@ttSX ........ ..... . 888t ...@StXtXt%St%@ttSSS%;;t;t%tttt%%%%;tSXtXt; 888888888 .%t8XtSS%SS;S;t%%tXtt@:t .%t.... .....: t8X% :.:;S;ttt%tttStS@XXXS@tXtt%S%S;tt;;;tttttX. t8888888888 %8;StSt%X;%%tt%%;%StXS@;;S888S8%8%;..88888X ..%.;%tS%S;tX;Sttt%XtXtt@S@XX%XtXXtSttttt;:X: 8@888t8t88X8;tXtX%SX;%t%tt%Xttttt%%%X%S%t%t%8X8 88t8 t..8.;%Stt;t%tt%%@tS;%%Xt%X;;t%XtSt@SXXX8888@888t88888:;;S8tS%XtXStS%t;t%t%%%%SXtSXt%@%StXS%tt888S8%8;.8X;%;tttt%%S;%;%tSt%tXt;tXtSt%%%%%;t8@8888Xt8SX@8@88888SSS%ttt%X;%tt;ttttXttt%%t%%tt%t%tX%@8888888:.St.t;;tt%ttttttttt%%t;;S;tSt%%%S%%X;8X888888XX8888@@tt8t8;Xt%St%Xt;;t;ttt;%t%%S%%tt%%St%;t@@8@8X8S :@8@:;%tt%;;t%tt%%%ttttt%tttttSStS;%%888X888@88888S8X88@@S%X%t%Xt%%t;:;t;ttttt%t%%%tStt%%%t;@@@tS%8 .888:8:;;tt%t;t;t;;tt%%ttttt%tttt%St%SX8XX88@8XX8@88X8@8t@tStS;%Xt%t;;;;;;;;tttttt%t;;tt%tt;%@.X8.%;%%t@@S X.:;;tt;tttttt;t;tt%t%S%%%t%%ttt;;;%@%@8@X8X8X88tS;ttt;t%%Xt%t;;t;;;t;t;ttttttttt%t%;8t8 @SSt%%%S%%;%8.@:;;ttt;tt%;ttt%t%S;%%%SS;t;:;;;;@88X8@88@%S.XSttt%tXt%%SX;ttt%t;t;t;tt;;;tt;tt%ttS8.;;;;:;8 8: 8:8 8.::;;;;t;;t;t;t;t;t%%%X;::;;;;;tt%S%t:. :.8X8StttSt;tSSttt%SSt%X;%%St%tttttttt%tt;::St@tXtS 8 8t .;t;;;:;::::;:;;;;tt;;tSt;:;;;;;;;8@8 X.;:;%X@tt:;ttt%%%tStX%X;Xt%X%XtXXSX%@%Xt%tttt;t@%XSSS8S88XX88%t;ttt;;;;::;:;:;;;;tt;tt;;;;;;;;@ %:..;;;:.:@ 8;;%%%S;%t%%%S;%%@t%SXt%%XtXSSS@SXXSS%@XXt@SSX8@8@@X8t;;tttt;t;;;;;t;;;;:;:;;;;;:;;tt;% 88S.%:;X888:8S;tX;t%t%@%X;%XSt%XttX%S%@;tS;tX;StXSt%@%%Xt;t%tt%%8;ttt;;t;t;tt;tttttt;tt;t;tt;;t;t:@%88888 SXXX;88tt%;tt%S;t8888888888888888888888888888888888

Algorithmic ‘war-story’ I Professor Latanya Sweeney and Google AdSense

• It’s 2013, and Sweeney, a researcher at Harvard University, investigated the delivery of targeted adverts by Google AdSense using a sample of racially associated names.

• First names associated predictively with non-white racial origin (such as DeShawn, Darnell and Jermaine) generated a far higher percentage of adverts associated with or using the word “arrest” when compared to ads delivered to “white” first names.

.X88XX 8 8 8 8 8 8 8 S. . . . . . . . . . . . ..8X 88@88888888888888 8;.. . . . . . . .. . . . :8 8 8888888888888888X8; . . .. . . . . . . . 8 8%8888@8@88888@88@88X:. . . . . .

. . . . S 888888888888888888888S . . . .. . . . . . . . S 8@@888888888888@888888 . . . . . .. . . . . . . :888888888888888888888SS. . . . .. . . . .8 888888888888888888888 . . . . . .

. . . . . . . .8 88888888888888888X8; . . .. . . . . . % 8@@888888888888888888. . . . . . . . .

. . . . .;888888888888888888888 %. . . .. . . . . . :8@8888888888888888888SX. . . . . . .

. . . . . .8 8888888888888@888888: . . . . . . . .. . . . . . S 88X88888888888888888@. . . . . .

. . . % 8@8888888888888888888. . . . . . .. . . . . . .:8@88888888X888888888888t;;;;;;t%SXX@888X; . .

. . . .8 88@8888888888888888X@S888@888888888888888t . .. . . . . ..8 @@88888888888888888%X88@8888X88888888888X .. .

. . % 88X88888888@88888S88S88S8X@ 8S88888X8X@XX @ . . .. . . . . . ;8@888888888@8888888@8S88S8888@ 8X8S88888X8 8:.. . .

. . :8888888888@8888888888S%888888888 8 88888888@. . . .. . . . . .8 88888888888888888888@88@88888888%88@88X8888. . . .

. . . S 8X888888888888888888888@8X8X88888S@8888 @@. . .. . . % 8X88@88888@8888888888888888888888%88888888 . . .

. . . . :88@888888888888888888%8 88888X8X8S8888S8 8S8 . . .. . ..8 8888888888888888888S.:X@@ X888888 88%8S888X; . . .

. . . .8 8X88888888888888X8X888 88888@888@@ 8S88S8;.. . . .. . % @@888888888888888888@8 88888@88X8S8X8 88@8 . . .

. . . .:88@888888888888888888@88@ 88@8888888X@S88888% . . . . .. . :8XX888888888888888888X88@88 8 888X8X88 8 888@ . . . .

. . .8 8@88888888888888888@8@88@888888888S888888. . . . .. . 8@ 8888888888X@X8X@S@:%8888888888888X88%%88X . . . . .

. . ;8S;:;t8X:. . . . :;t:::;:;.... ... @8@ . .. .8S8X : %88t: . . . . . .. . ;88: . . . . . .

. ;:@ 8:. :X%8: . . . . . . . .t88;.: 88. . .. .@% ;8: @t %X. t8 t@8;t t8% ;8. X% . .

. S@t: ..::::::: .S8X%8% 8 : :.8;X8;X88; SS8 88 88@S ;88:; 8 X8.8SX .. X@;.. 8%ttt%S;;88 %...%;8t.t8S:.. % 88; 88t;::88@S t8X.:8.St%t@@.: .

. ;:@8 . ...::.8@ t.88. SXtt%St .88.@8 ; .: @X t88tS8 t .88SS . ; 8X %8t . 88 :.88 ..;8%tXSt . t%@S.t8S. 88.. ..:

. . ..8tX:8XSS8S%XX: %;@%X%SXSS@ 8S%8StXt@ t.X@ 8S%8 8@S t88:8X%8X%X;S@ . ...SS.::: t8: .8t ; .8; ;8:.;.t8: S% ; :8 8S 8 8..;@ :: tX. .

. . . . . . .. . . %@%. . SXt;.. . . .. . . . . . . . . 8X%%88t%S8 . . .

. ;XS. . . . tX; .;@888%. . :8X;;X@:. . . . . . ...8@@ . . . ;88 8S88888t:. . . . . .

. 8@tX.: . ;88.@X%8t;S;S@ . . . . . . . . . .%S%tX8%. . .%8SS8tt88 88:%. .. . :@@%S8% . %8%%8%X8; ;8SSX8: . .X8SS8S..t8X ;%XS .X88t:SX;88.:.8@:8; ; 8X:;@@@. S8;Xt:X8tt XS8t.%8;t ::8@:;@8%.XXt; 88:. .@8 S..;@X88 :8t:SS.@; ;.8X;..S 88.S8S@..;.8S t8::.:88;::8%t..t:8S;;8t:...@8@ ;%Xt t88 . .%S@tt88@;:;::tX:.S8S . 88. t;%%;.SX X@@;:;;:tStX8tt;;t;:@%t:tXS . :88 8;t . . 88@8.;:::;::. %8S .88. .%8:@88S@tt:::::::S%XS .t8%.@8 S..;@X88.%8.@:.;8XS%: 8S;..t%.S S8@. .88;.8:8;..t8;.:8tt..tX 8.S8 . . ;.%X @.XSSt;;S8. @.SXXX%%% :8:XSSS.8: @S8 .8St ;S%S%S%%; .8.XSSS:8.

Algorithmic ‘war-story’ I Professor Latanya Sweeney and Google AdSense

Was a “decision” taken with reference to Sweeney?

• No effect on legal status (public status, such as citizenship, or private status, like capacity to make a will)

• A so-called racial group was impugned by assumption of above average criminality: takes us to a sort of ‘group right’, very different from individual liberal paradigm rights granted by the GDPR.

• Even given impact on Sweeney as an individual constructed through group membership, was it “significant”?

Algorithmic ‘war-story’ II “Jew Watch”

• In 2004, the Google search algorithm(s) placed a site “Jew Watch” at the top of the rankings for many people who searched on the word “Jew”. Google pulled what we now might call the “neutrality” defence, or fallacy.

• Was there a significant decision made? Only individuals searching “Jew” could be offended. While in algorithmic defamation cases, searched names are clearly linked to individuals, here, the searched term is linked broadly to a group.

The pardoxes of GDPR, art 22 Automated individual decision-making, including profilingWhen sensitive data, such as so-called race, is being processed, you do not have a right to have a human-in-the-loop if - Explicit consent was given (or substantial public interest in basis of MS law); - Safeguards in Recital 71 are in place, which include

Under art 22, it seems possible to infer that where you have no primary right to object, you have a binding right to explanation. Else, no. Paradoxical, in ways. To trigger such a right, you need to know that sensitive data — in the war stories, inferred sensitive data — were being processed. How, without a right to explanation?

Very shaky grounds to found EU transparency rights upon.* * For more, including an analysis of case law, see Wachter et al. (forthcoming) Why a right to explanation does not exist in the General Data Protection Regulation, International Data Privacy Law

Continuing our quest, with GDPR, art 15? Right of access by the data subject

Right of access: Article 15 — which existed in the DPD as article 12(a) — provides that the data subject shall have the right to confirm whether or not personal data relating to him or her are being processed by a controller and if that is the case, access to that personal data and information including, in the context of automated decision making, “meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing” (art 15(1)(h)).

It has its own problems in the form of a carve out for the protection of trade secrets and intellectual property, although Recital 63 now counsels that this should not justify “a refusal to provide all information to the data subject” (emph added)

How far must we breach IP for our beloved explanations? A trip to old computer science literature.

• Early work on explanations emphasised that ‘following the logic’ is not useful; need to optimise explanations for people.

• Can do this by modelling a model, with humans in mind, to replicate core logics • ‘What would I have need to have done…’ • ‘What factor was most important…’ • ‘Who got the same result to me?’ • What results did people like me in X way get?

• Arguably this makes IP less of a concern.

simpler model!!!

*Wick and Thompson (1992) (doi:10.1016/0004-3702(92)90087-E); **Tickle et al. (1998) (doi:10.1109/72.728352)

And yes, this works today*

*Ribeiro et al. (2016) (doi:10.1145/2939672.2939778). Decompositional methods might overtake the effectiveness of this. See Montavon et al. (2017) (doi:10.1145/2939672.2939778)

What might scupper our rights to explanation?

Domain Easy to explain an algorithm if there are only a few inputs. But currently they take many! Sensors, location, browsing history; and many of them are abstract.

Users Models of models simplify the complex decision system to make it more explainable. Who do they simplify away? Probably the ‘weirdos’ that need the law most.

Source: doi: 10.1145/2976749.2978392

Other rights, other remedies

Right to erasure: Be ‘forgotten’ from a dataset. But, are you removed from a model? Are people like you removed from a model? More and more things sensitive data: studies showing you can predict depression from smartphone habits, for example

Right to portability: Take your data with you. Take inferences about you too? Article 29 Working Party think not in draft guidance; unclear though.

Data Protection Impact Assessments: High risk processing activities require pre-emptive analysis, report to Data Protection Authority in certain cases.

Data Protection by Design: Use technological/organsiational means to avoid harms

Data Protection Certification: Countries can establish certification systems

Source: doi: 10.1145/2976749.2978392

thanks — q? Paper forthcoming in Duke Law and Technology Review

Download! https://ssrn.com/abstract=2972855

lilian edwards // @lilianedwards lilian.edwards@strath.ac.uk

michael veale // @mikarv m.veale@ucl.ac.uk