1

TheNooscopeManifested

ArtificialIntelligenceasInstrumentofKnowledgeExtractivism

MatteoPasquinelliandVladanJoler

1. Someenlightenmentregardingtheprojecttomechanisereason.2. Theassemblylineofmachinelearning:Data,Algorithm,Model.3. Thetrainingdataset:thesocialoriginsofmachineintelligence.4. ThehistoryofAIastheautomationofperception.5. Thelearningalgorithm:compressingtheworldintoastatisticalmodel.6. Allmodelsarewrong,butsomeareuseful.7. Worldtovector.8. Thesocietyofclassificationandpredictionbots.9. Faultsofastatisticalinstrument:theundetectionofthenew.10. Adversarialintelligencevs.statisticalintelligence.11. LabourintheageofAI.

Full citation: Matteo Pasquinelli and Vladan Joler, “The Nooscope Manifested: Artificial Intelligence as Instrument of Knowledge Extractivism”, KIM research group (Karlsruhe University of Arts and Design) and Share Lab (Novi Sad), 1 May 2020 (preprint forthcoming for AI and Society). https://nooscope.ai

2

1.Someenlightenmentregardingtheprojecttomechanisereason.TheNooscopeisacartographyofthelimitsofartificialintelligence,intendedasaprovocationtobothcomputer science and the humanities. Anymap is a partial perspective, a way to provoke debate.Similarly,thismapisamanifesto—ofAIdissidents.Itsmainpurposeistochallengethemystificationsofartificial intelligence.First,asatechnicaldefinitionof intelligenceand,second,asapolitical formthatwouldbeautonomousfromsocietyandthehuman.1Intheexpression‘artificialintelligence’theadjective‘artificial’carriesthemythofthetechnology’sautonomy:ithintstocaricatural‘alienminds’that self-reproduce in silicobut, actually,mystifies twoprocessesof proper alienation: the growinggeopoliticalautonomyofhi-techcompaniesandtheinvisibilizationofworkers’autonomyworldwide.The modern project to mechanise human reason has clearly mutated, in the 21st century, into acorporate regime of knowledge extractivism and epistemic colonialism.2This is unsurprising, sincemachinelearningalgorithmsarethemostpowerfulalgorithmsforinformationcompression.

ThepurposeoftheNooscopemapistosecularizeAIfromtheideologicalstatusof‘intelligentmachine’tooneofknowledgeinstrument.Ratherthanevokinglegendsofaliencognition,it ismorereasonabletoconsidermachinelearningasaninstrumentofknowledgemagnificationthathelpstoperceivefeatures,patterns,andcorrelationsthroughvastspacesofdatabeyondhumanreach.Inthehistoryofscienceandtechnology,thisisnonews:ithasalreadybeenpursuedbyopticalinstrumentsthroughoutthehistoriesofastronomyandmedicine.3Inthetraditionofscience,machinelearningisjustaNooscope,aninstrumenttoseeandnavigatethespaceofknowledge(fromtheGreekskopein‘toexamine,look’andnoos‘knowledge’).

BorrowingtheideafromGottfriedWilhelmLeibniz,theNooscopediagramappliestheanalogyof optical media to the structure of all machine learning apparatuses. Discussing the power of hiscalculusratiocinatorand‘characteristicnumbers’(theideatodesignanumericaluniversallanguagetocodifyandsolvealltheproblemsofhumanreasoning),Leibnizmadeananalogywithinstrumentsofvisualmagnificationsuchasthemicroscopeandtelescope.Hewrote:‘Oncethecharacteristicnumbersareestablishedformostconcepts,mankindwillthenpossessanewinstrumentwhichwillenhancethecapabilitiesofthemindtoafargreaterextentthanopticalinstrumentsstrengthentheeyes,andwillsupersede themicroscope and telescope to the same extent that reason is superior to eyesight.’4Althoughthepurposeofthistextisnottoreiteratetheoppositionbetweenquantitativeandqualitativecultures, Leibniz’s credo need not be followed. Controversies cannot be conclusively computed.Machinelearningisnottheultimateformofintelligence.

Instrumentsofmeasurementandperceptionalwayscomewithinbuiltaberrations.Inthesameway that the lenses ofmicroscopes and telescopes are never perfectly curvilinear and smooth, thelogical lenses of machine learning embody faults and biases. To understandmachine learning andregisteritsimpactonsocietyistostudythedegreebywhichsocialdataarediffractedanddistortedbytheselenses.ThisisgenerallyknownasthedebateonbiasinAI,butthepoliticalimplicationsofthelogicalformofmachinelearningaredeeper.Machinelearningisnotbringinganewdarkagebutoneofdiffractedrationality,inwhich,asitwillbeshown,anepistemeofcausationisreplacedbyoneofautomatedcorrelations.Moreingeneral,AIisanewregimeoftruth,scientificproof,socialnormativityandrationality,whichoftendoestaketheshapeofastatisticalhallucination.Thisdiagrammanifestoisanotherway to say thatAI, thekingof computation (patriarchal fantasyofmechanisedknowledge,‘masteralgorithm’andalphamachine)isnaked.Here,wearepeepingintoitsblackbox.

3

Ontheinventionofmetaphorsasinstrumentofknowledgemagnification.EmanueleTesauro,Ilcanocchialearistotelico[TheAristotelianTelescope],frontispieceofthe1670edition,Turin.

2.Theassemblylineofmachinelearning:Data,Algorithm,Model.ThehistoryofAI isahistoryofexperiments,machinefailures,academiccontroversies,epicrivalriesaroundmilitaryfunding,popularlyknownas‘wintersofAI.’5AlthoughcorporateAItodaydescribesitspowerwiththelanguageof‘blackmagic’and‘superhumancognition’,currenttechniquesarestillattheexperimentalstage.6AIisnowatthesamestageaswhenthesteamenginewasinvented,beforethelawsofthermodynamicsnecessarytoexplainandcontrolitsinnerworkings,hadbeendiscovered.Similarly, today, thereareefficientneuralnetworks for image recognition,but there isno theoryoflearningtoexplainwhytheyworksowellandhowtheyfailsobadly.Likeanyinvention,theparadigmofmachinelearningconsolidatedslowly,inthiscasethroughthelasthalf-century.Amasteralgorithmhasnotappearedovernight.Rather,therehasbeenagradualconstructionofamethodofcomputationthatstillhastofindacommonlanguage.Manualsofmachinelearningforstudents,forinstance,donotyetshareacommonterminology.Howtosketch,then,acriticalgrammarofmachinelearningthatmaybeconciseandaccessible,withoutplayingintotheparanoidgameofdefiningGeneralIntelligence?

As an instrumentof knowledge,machine learning is composedof anobject tobeobserved(training dataset), an instrument of observation (learning algorithm) and a final representation(statisticalmodel).Theassemblageofthesethreeelementsisproposedhereasaspuriousandbaroquediagram ofmachine learning, extravagantly termed Nooscope.7Staying with the analogy of opticalmedia,theinformationflowofmachinelearningis likealightbeamthatisprojectedbythetrainingdata,compressedbythealgorithmanddiffractedtowardstheworldbythelensofthestatisticalmodel.

4

TheNooscopediagramaimstoillustratetwosidesofmachinelearningatthesametime:howitworksandhowitfails—enumeratingitsmaincomponents,aswellasthebroadspectrumoferrors,limitations,approximations,biases,faults,fallaciesandvulnerabilitiesthatarenativetoitsparadigm.8This double operation stresses that AI is not a monolithic paradigm of rationality but a spuriousarchitecturemadeofadaptingtechniquesandtricks.Besides,thelimitsofAIarenotsimplytechnicalbutareimbricatedwithhumanbias. IntheNooscopediagramtheessentialcomponentsofmachinelearningarerepresentedatthecentre,humanbiasesandinterventionsontheleft,andtechnicalbiasesand limitations on the right. Optical lenses symbolize biases and approximations representing thecompressionanddistortionoftheinformationflow.Thetotalbiasofmachinelearningisrepresentedbythecentrallensofthestatisticalmodelthroughwhichtheperceptionoftheworldisdiffracted.

The limitations of AI are generally perceived today thanks to the discourse on bias —theamplificationofgender,race,ability,andclassdiscriminationbyalgorithms.Inmachinelearning,itisnecessarytodistinguishbetweenhistoricalbias,datasetbias,andalgorithmbias,allofwhichoccuratdifferentstagesoftheinformationflow.9Historicalbias(orworldbias)isalreadyapparentinsocietybefore technological intervention. Nonetheless, the naturalisation of such bias, that is the silentintegration of inequality into an apparently neutral technology is by itself harmful.10ParaphrasingMichelle Alexander, Ruha Benjamin has called it the New Jim Code: ‘the employment of newtechnologiesthatreflectandreproduceexistinginequalitiesbutthatarepromotedandperceivedasmore objective or progressive than the discriminatory systems of a previous era.’11Dataset bias isintroducedthroughthepreparationoftrainingdatabyhumanoperators.Themostdelicatepartoftheprocessisdatalabelling,inwhicholdandconservativetaxonomiescancauseadistortedviewoftheworld,misrepresenting social diversities and exacerbating social hierarchies (see below the case ofImageNet).

Algorithmic bias (also known as machine bias, statistical bias or model bias, to which theNooscopediagramgivesparticularattention)isthefurtheramplificationofhistoricalbiasanddatasetbias bymachine learning algorithms. The problem of bias hasmostly originated from the fact thatmachine learning algorithms are among the most efficient for information compression, whichengendersissuesofinformationresolution,diffractionandloss.12Sinceancienttimes,algorithmshavebeenproceduresofaneconomicnature,designedtoachievearesultintheshortestnumberofstepsconsuming the least amount of resources: space, time, energy and labour. 13The arms race of AIcompanies is, still today, concernedwith finding the simplest and fastest algorithmswithwhich tocapitalisedata.IfinformationcompressionproducesthemaximumrateofprofitincorporateAI,fromthesocietalpointofview,itproducesdiscriminationandthelossofculturaldiversity.

While the social consequences of AI are popularly understood under the issue of bias, thecommonunderstandingoftechnicallimitationsisknownastheblackboxproblem.Theblackboxeffectisanactualissueofdeepneuralnetworks(whichfilterinformationsomuchthattheirchainofreasoningcannot be reversed) but has becomea generic pretext for theopinion thatAI systems are not justinscrutableandopaque,buteven‘alien’andoutofcontrol.14Theblackboxeffectispartofthenatureofanyexperimentalmachineattheearlystageofdevelopment(ithasalreadybeennoticedthatthefunctioningofthesteamengineremainedamysteryforsometime,evenafterhavingbeensuccessfullytested). The actual problem is the black box rhetoric, which is closely tied to conspiracy theorysentimentsinwhichAIisanoccultpowerthatcannotbestudied,known,orpoliticallycontrolled.

5

3.Thetrainingdataset:thesocialoriginsofmachineintelligence.Massdigitalisation,whichexpandedwiththeInternetinthe1990sandescalatedwithdatacentresinthe2000s,hasmadeavailablevast resourcesofdata that, for the first time inhistory,are freeandunregulated.Aregimeofknowledgeextractivism(thenknownasBigData)graduallyemployedefficientalgorithmstoextract‘intelligence’fromtheseopensourcesofdata,mainlyforthepurposeofpredictingconsumerbehavioursandsellingads.Theknowledgeeconomymorphedintoanovelformofcapitalism,calledcognitivecapitalismandthensurveillancecapitalism,bydifferentauthors.15ItwastheInternetinformationoverflow,vastdatacentres, fastermicroprocessorsandalgorithmsfordatacompressionthatlaidthegroundworkfortheriseofAImonopoliesinthe21stcentury.

WhatkindofculturalandtechnicalobjectisthedatasetthatconstitutesthesourceofAI?Thequalityoftrainingdataisthemostimportantfactoraffectingtheso-called‘intelligence’thatmachinelearning algorithms extract. There is an important perspective to take into account, in order tounderstandAIasaNooscope.Dataisthefirstsourceofvalueandintelligence.Algorithmsaresecond;theyarethemachinesthatcomputesuchvalueandintelligenceintoamodel.However,trainingdataarenever raw, independentandunbiased (theyarealready themselves ‘algorithmic’).16Thecarving,formattingandeditingoftrainingdatasetsisalaboriousanddelicateundertaking,whichisprobablymoresignificantforthefinalresultsthanthetechnicalparametersthatcontrolthelearningalgorithm.Theactofselectingonedatasourceratherthananotheristheprofoundmarkofhumaninterventionintothedomainofthe‘artificial’minds.

Thetrainingdatasetisaculturalconstruct,notjustatechnicalone.Itusuallycomprisesinputdatathatare associatedwith ideal output data, such as pictureswith their descriptions, also called labels ormetadata.17Thecanonicalexamplewouldbeamuseumcollectionanditsarchive, inwhichartworksareorganisedbymetadatasuchasauthor,year,medium,etc.Thesemioticprocessofassigninganameoracategorytoapictureisneverimpartial;thisactionleavesanotherdeephumanimprintonthefinalresultofmachinecognition.Atrainingdatasetformachinelearningisusuallycomposedthroughthefollowingsteps:1)production:labourorphenomenathatproduceinformation;2)capture:encodingofinformationintoadataformatbyaninstrument:3)formatting:organisationofdataintoadataset:4)labelling:insupervisedlearning,theclassificationofdataintocategories(metadata).

Machine intelligence is trained on vast datasets that are accumulated in ways neithertechnicallyneutralnorsociallyimpartial.Rawdatadoesnotexist,asitisdependentonhumanlabour,personaldata,andsocialbehaviours thataccrueover longperiods, throughextendednetworksandcontroversial taxonomies. 18 The main training datasets for machine learning (NMIST, ImageNet,LabelledFaces intheWild,etc.)originated incorporations,universities,andmilitaryagenciesof the

Metadata / Labels

Database format

Data

6

GlobalNorth.Buttakingamorecarefullook,onediscoversaprofounddivisionoflabourthatinnervatesintotheGlobalSouthviacrowdsourcingplatformsthatareusedtoeditandvalidatedata.19TheparableoftheImageNetdatasetexemplifiesthetroublesofmanyAIdatasets.ImageNetisatrainingdatasetforDeepLearningthathasbecomethedefactobenchmarkforimagerecognitionalgorithms:indeed,theDeepLearningrevolutionstartedin2012whenAlexKrizhevsky,IlyaSutskeverandGeoffreyHintonwontheannual ImageNetchallengewiththeconvolutionalneuralnetworkAlexNet.20ImageNetwasinitiatedbycomputerscientistFei-FeiLibackin2006.21Fei-FeiLihadthreeintuitionstobuildareliabledatasetforimagerecognition.First,todownloadmillionsoffreeimagesfromwebservicessuchasFlickrand Google. Second, to adopt the computational taxonomyWordNet for image labels.22Third, tooutsourcetheworkoflabellingmillionsofimagesviathecrowdsourcingplatformAmazonMechanicalTurk.Attheendoftheday(andoftheassemblyline),anonymousworkersfromallovertheplanetwerepaidfewcentspertasktolabelhundredsofpicturesperminuteaccordingtotheWordNettaxonomy:theirlabourresultedintheengineeringofacontroversialculturalconstruct.AIscholarsKateCrawfordand artist Trevor Paglen have investigated and disclosed the sedimentation of racist and sexistcategories in ImageNet taxonomy: see the legitimation of the category ‘failure, loser, nonstarter,unsuccessfulperson’forahundredarbitrarypicturesofpeople.23

ThevoraciousdataextractivismofAIhascausedanunforeseeablebacklashondigitalculture:intheearly2000s,LawrenceLessigcouldnotpredictthatthelargerepositoryofonlineimagescreditedby Creative Commons licenses would a decade later become an unregulated resource for facerecognitionsurveillancetechnologies.Insimilarways,personaldataiscontinuallyincorporatedwithouttransparency into privatised datasets for machine learning. In 2019 artist and AI researcher AdamHarveyforthefirsttimedisclosedthenonconsensualuseofpersonalphotosintrainingdatasetsforface recognition. Harvey’s disclosure caused Stanford University, Duke University andMicrosoft towithdrawtheirdatasetsamidstamajorprivacyinfringementscandal.24Onlinetrainingdatasetstriggerissuesofdatasovereigntyandcivilrightsthattraditional institutionsareslowtocounteract(seetheEuropean General Data Protection Regulation).25If 2012 was the year in which the Deep Learningrevolution began, 2019 was the year in which its sources were discovered to be vulnerable andcorrupted.

CombinatorialpatternsandKuficscripts,Topkapiscroll,ca.1500,Iran.

7

4.ThehistoryofAIastheautomationofperception.TheneedtodemystifyAI(atleastfromthetechnicalpointofview)isunderstoodinthecorporateworldtoo.HeadofFacebookAIandgodfatherofconvolutionalneuralnetworksYannLeCunreiteratesthatcurrentAIsystemsarenotsophisticatedversionsofcognition,butrather,ofperception.Similarly,theNooscope diagram exposes the skeleton of the AI black box and shows that AI is not a thinkingautomaton but an algorithm that performs pattern recognition. The notion of pattern recognitioncontainsissuesthatmustbeelaboratedupon.Whatisapattern,bytheway?Isapatternuniquelyavisual entity? What does itmean to read social behaviours as patterns? Is pattern recognition anexhaustive definition of intelligence? Most likely not. To clarify these issues, it would be good toundertakeabriefarchaeologyofAI.

ThearchetypemachineforpatternrecognitionisFrankRosenblatt’sPerceptron. Inventedin1957atCornellAeronauticalLaboratoryinBuffalo,NewYork,itsnameisashorthandfor‘PerceivingandRecognizingAutomaton.’26Givenavisualmatrixof20x20photoreceptors,thePerceptroncanlearnhowtorecognisesimpleletters.Avisualpatternisrecordedasanimpressiononanetworkofartificialneurons thatare firingup in concertwith the repetitionof similar imagesandactivatingone singleoutputneuron.Theoutputneuron fires1=true, ifagiven image is recognised,or0=false, ifagivenimageisnotrecognised.

Theautomationofperception,asavisualmontageofpixelsalongacomputationalassemblyline, was originally implicit McCulloch and Pitt’s concept of artificial neural networks. 27 Once thealgorithmforvisualpatternrecognitionsurvivedthe‘winterofAI’andprovedefficientinthelate2000s,it was applied also to non-visual datasets, properly inaugurating the age of Deep Learning (theapplicationofpatternrecognitiontechniquestoallkindsofdata,notjustvisual).Today,inthecaseofself-drivingcars,thepatternsthatneedtoberecognisedareobjectsinroadscenarios.Inthecaseofautomatic translation, thepatterns thatneed tobe recognisedare themost commonsequencesofwordsacrossbilingualtexts.Regardlessoftheircomplexity,fromthenumericalperspectiveofmachinelearning,notionssuchas image,movement, form,style,andethicaldecisioncanallbedescribedasstatisticaldistributionsofpattern.Inthissense,patternrecognitionhastrulybecomeanewculturaltechnique that is used in various fields. For explanatory purposes, the Nooscope is described as amachinethatoperatesonthreemodalities:training,classification,andprediction. Inmore intuitiveterms,thesemodalitiescanbecalled:patternextraction,patternrecognition,andpatterngeneration.

Rosenblatt’sPerceptronwasthefirstalgorithmthatpavedthewaytomachinelearninginthecontemporarysense.Atatimewhen‘computerscience’hadnotyetbeenadoptedasdefinition,thefieldwascalled‘computationalgeometry’andspecifically‘connectionism’byRosenblatthimself.Thebusiness of theseneural networks, however,was to calculate a statistical inference.What a neuralnetworkcomputes,isnotanexactpatternbutthestatisticaldistributionofapattern.JustscrapingthesurfaceoftheanthropomorphicmarketingofAI,onefindsanothertechnicalandculturalobjectthatneedsexamination:thestatisticalmodel.Whatisthestatisticalmodelinmachinelearning?Howisitcalculated?Whatistherelationshipbetweenastatisticalmodelandhumancognition?Thesearecrucialissuestoclarify.Intermsoftheworkofdemystificationthatneedstobedone(alsotoevaporatesomenaïvequestions), itwouldbegoodtoreformulatethetritequestion‘Canamachinethink?’ intothetheoretically sounder questions ‘Can a statistical model think?’, ‘Can a statistical model developconsciousness?’,etcetera.

8

5.Thelearningalgorithm:compressingtheworldintoastatisticalmodel.

The algorithms of AI are often evoked as alchemic formulas, capable of distilling ‘alien’ forms ofintelligence. But what do the algorithms of machine learning really do? Few people, including thefollowersofAGI(ArtificialGeneralIntelligence),bothertoaskthisquestion.Algorithmisthenameofaprocess, whereby a machine performs a calculation. The product of such machine processes is astatistical model (more accurately termed an ‘algorithmic statistical model’). In the developercommunity, theterm ‘algorithm’ is increasingly replacedwith ‘model.’This terminologicalconfusionarisesfromthefactthatthestatisticalmodeldoesnotexistseparatelyfromthealgorithm:somehow,the statistical model exists inside the algorithm under the form of distributed memory across itsparameters.Forthesamereason,itisessentiallyimpossibletovisualiseanalgorithmicstatisticalmodel,asisdonewithsimplemathematicalfunctions.Still,thechallengeisworthwhile.

Inmachinelearning,therearemanyalgorithmarchitectures:simplePerceptron,deepneuralnetwork,SupportVectorMachine,Bayesiannetwork,Markovchain,autoencoder,Boltzmannmachine,etc. Each of these architectures has a different history (often rooted in military agencies andcorporationsoftheGlobalNorth).Artificialneuralnetworksstartedassimplecomputingstructuresthatevolvedintocomplexoneswhicharenowcontrolledbyafewhyperparametersthatexpressmillionsof parameters. 28 For instance, convolutional neural networks are described by a limited set ofhyperparameters (numberof layers,numberofneuronsper layer, typeofconnection,behaviourofneurons, etc.) that project a complex topology of thousands of artificial neurons with millions ofparameters in total. Thealgorithmstartsasablank slateand,during theprocess called training,or‘learningfromdata',adjustsitsparametersuntilitreachesagoodrepresentationoftheinputdata.Inimage recognition,asalreadyseen, thecomputationofmillionsofparametershas to resolve intoasimplebinaryoutput:1=true,agivenimageisrecognised;or0=false,agivenimageisnotrecognised.29

Source:https://www.asimovinstitute.org/neural-network-zoo

Perceptron (P)

Feed Forward (FF)

Radial Basis Networks (RBF) Deep Feed Forward (DFF) Recurrent Neural Network (RNN) Support Vector Machine (SVM)

Auto Encoder (AE) Variational AE (VAE) Sparse AE (SAE)

Markov Chain (MC)Hopfield Network (HN)

Boltzmann Machine (BM)Restricted BM (RBM)

Deep Belief Network (DBN) Generative Adverserial Network (GAN) Deconvolutional Network (DN)

Deep Convolutional Inverse Graphics Network (DCIGN) Generative Adversarial Network (GAN) Echo State Network (ESN)

Neural Turing Machine (NTM) Deep Residual Network (DRN) Kohonen Network (KN)

9

Attemptinganaccessibleexplanationof therelationshipbetweenalgorithmandmodel, let’shavealookatthecomplexInceptionv3algorithm,adeepconvolutionalneuralnetworkforimagerecognitiondesignedatGoogleandtrainedontheImageNetdataset.Inceptionv3issaidtohavea78%accuracyinidentifying the label of a picture, but the performanceof ‘machine intelligence’ in this case can bemeasuredalsobytheproportionbetweenthesizeoftrainingdataandthetrainedalgorithm(ormodel).ImageNetcontains14millionimageswithassociatedlabelsthatoccupyapproximately150gigabytesofmemory.Ontheotherhand,Inceptionv3,whichismeanttorepresenttheinformationcontainedinImageNet,isonly92megabytes.Theratioofcompressionbetweentrainingdataandmodelpartiallydescribesalsotherateofinformationdiffraction.AtablefromtheKerasdocumentationcomparesthesevalues(numbersofparameters,layerdepth,filedimensionandaccuracy)forthemainmodelsofimagerecognition.30Thisisabrutalistbuteffectivewaytoshowtherelationbetweenmodelanddata,toshowhowthe‘intelligence’ofalgorithmsismeasuredandassessedinthedevelopercommunity.

Statisticalmodelshavealwaysinfluencedcultureandpolitics.Theydidnotjustemergewithmachinelearning:machinelearningisjustanewwaytoautomatethetechniqueofstatisticalmodelling.WhenGretaThunbergwarns‘Listentoscience.’whatshereallymeans,beingagoodstudentofmathematics,is ‘Listen to the statisticalmodels of climate science.’ No statisticalmodels, no climate science: noclimatescience,noclimateactivism.Climatescienceisindeedagoodexampletostartwith,inordertounderstandstatisticalmodels.GlobalwarminghasbeencalculatedbyfirstcollectingavastdatasetoftemperaturesfromEarth’ssurfaceeachdayoftheyear,andsecond,byapplyingamathematicalmodelthat plots the curve of temperature variations in the past and projects the same pattern into thefuture. 31Climate models are historical artefacts that are tested and debated within the scientificcommunity, and today, also beyond.32Machine learningmodels, on the contrary, are opaque andinaccessible to community debate. Given the degree of myth-making and social bias around itsmathematicalconstructs,AIhasindeedinauguratedtheageofstatisticalsciencefiction.Nooscopeistheprojectorofthislargestatisticalcinema.

10

6.Allmodelsarewrong,butsomeareuseful.‘Allmodelsarewrong,butsomeareuseful’—thecanonicaldictumoftheBritishstatisticianGeorgeBox has long encapsulated the logical limitations of statistics and machine learning.33This maxim,however,isoftenusedtolegitimisethebiasofcorporateandstateAI.Computerscientistsarguethathumancognitionreflectsthecapacitytoabstractandapproximatepatterns.Sowhat’stheproblemwithmachinesbeingapproximate,anddoingthesame?Withinthisargument,itisrhetoricallyrepeatedthat‘themapisnottheterritory’.Thissoundsreasonable.ButwhatshouldbecontestedisthatAIisaheavilycompressedanddistortedmapoftheterritoryandthatthismap,likemanyformsofautomation,isnotopentocommunitynegotiation.AIisamapoftheterritorywithoutcommunityaccessandcommunityconsent.34

Howdoesmachinelearningplotastatisticalmapoftheworld?Let’sfacethespecificcaseofimagerecognition(thebasicformofthelabourofperception,whichhasbeencodifiedandautomatedaspatternrecognition).35Givenanimagetobeclassified,thealgorithmdetectstheedgesofanobjectas the statistical distribution of dark pixels surrounded by light ones (a typical visual pattern). Thealgorithmdoesnotknowwhatanimageis,doesnotperceiveanimageashumancognitiondoes,itonlycomputespixels,numericalvaluesofbrightnessandproximity.Thealgorithmisprogrammedtorecordonlythedarkedgeofaprofile(thatistofitthatdesiredpattern)andnotallthepixelsacrosstheimage(thatwouldresultinoverfittingandrepeatingthewholevisualfield).Astatisticalmodelissaidtobetrainedsuccessfullywhenitcanelegantlyfitonlytheimportantpatternsofthetrainingdataandapplythosepatternsalsotonewdata‘inthewild’.Ifamodellearnsthetrainingdatatoowell,itrecognisesonlyexactmatchesoftheoriginalpatternsandwilloverlookthosewithclosesimilarities,‘inthewild’.Inthiscase,themodelisoverfitting,becauseithasmeticulouslylearnteverything(includingnoise)andisnotabletodistinguishapatternfromitsbackground.Ontheotherhand,themodelisunderfittingwhenitisnotabletodetectmeaningfulpatternsfromthetrainingdata.Thenotionsofdataoverfitting,fittingandunderfittingcanbevisualisedonaCartesianplane.

X Statistical Anomaly

X X

X

XX

Underfitting

Fitting

Overfitting

Interpolation and ExtrapolationApproximation Model fitting

ExtrapolationInterpolationExtrapolation

11

Thechallengeofguardingtheaccuracyofmachinelearninglaysincalibratingtheequilibriumbetweendataunderfittingandoverfitting,whichisdifficulttodobecauseofdifferentmachinebiases.Machinelearning isa termthat,asmuchas ‘AI',anthropomorphizesapieceof technology:machine learninglearns nothing in theproper senseof theword, as a humandoes;machine learning simplymaps astatistical distribution of numerical values and draws a mathematical function that hopefullyapproximateshumancomprehension.Thatbeingsaid,machinelearningcan,forthisreason,castnewlightonthewaysinwhichhumanscomprehend.

Thestatisticalmodelofmachinelearningalgorithmsisalsoanapproximationinthesensethatitguessesthemissingpartsofthedatagraph:eitherthroughinterpolation,whichisthepredictionofanoutputywithintheknownintervaloftheinputxinthetrainingdataset,orthroughextrapolation,whichisthepredictionofoutputybeyondthelimitsofx,oftenwithhighrisksof inaccuracy.This iswhat ‘intelligence’ means today within machine intelligence: to extrapolate a non-linear functionbeyondknowndataboundaries.AsDanMcQuillianaptlyputs it: ‘There isno intelligence inartificialintelligence, nor does it learn, even though its technical name is machine learning, it is simplymathematicalminimization.’36

Itisimportanttorecallthatthe‘intelligence’ofmachinelearningisnotdrivenbyexactformulasofmathematicalanalysis,butbyalgorithmsofbruteforceapproximation.Theshapeofthecorrelationfunctionbetween inputx andoutputy is calculatedalgorithmically, stepby step, through tiresomemechanicalprocessesofgradualadjustment(likegradientdescent,forinstance)thatareequivalenttothedifferentialcalculusofLeibnizandNewton.Neuralnetworksaresaidtobeamongthemostefficientalgorithmsbecausethesedifferentialmethodscanapproximatetheshapeofanyfunctiongivenenoughlayersofneuronsandabundantcomputingresources.37Brute-forcegradualapproximationofafunctionisthecorefeatureoftoday’sAI,andonlyfromthisperspectivecanoneunderstanditspotentialitiesand limitations— particularly its escalating carbon footprint (the training of deep neural networksrequiresexorbitantamountsofenergybecauseofgradientdescentandsimilartrainingalgorithmsthatoperateonthebasisofcontinuousinfinitesimaladjustments).38

Multidimensionalvector space

Black box horizon

4

3

2

1

4321

43

21

engine

sky

wings

helicopter (0,2,4)

drone (0,3,3)

rocket (0,4,2)eagle (3,0,3)

bee (3,0,2)

goose (2,0,4)

jet (1,1,1)

12

7.Worldtovector.The notions of data fitting, overfitting, underfitting, interpolation and extrapolation can be easilyvisualisedintwodimensions,butstatisticalmodelsusuallyoperatealongmultidimensionalspacesofdata.Beforebeinganalysed,dataareencodedintoamulti-dimensionalvectorspacethatisfarfromintuitive.Whatisavectorspaceandwhyisitmulti-dimensional?Cardon,CointetandMazièredescribethevectorialisationofdatainthisway:

Aneuralnetworkrequirestheinputsofthecalculatortotakeontheformofavector.Therefore,theworldmustbecodedinadvanceintheformofapurelydigitalvectorialrepresentation.Whilecertainobjectssuchas imagesarenaturallybrokendown intovectors,otherobjectsneedtobe ‘embedded’withinavectorialspacebeforeitispossibletocalculateorclassifythemwithneuralnetworks.Thisisthecaseoftext,whichistheprototypicalexample.Toinputawordintoaneuralnetwork,theWord2vectechnique ‘embeds’ it into a vectorial space thatmeasures its distance from the otherwords in thecorpus.Wordsthusinheritapositionwithinaspacewithseveralhundredsofdimensions.Theadvantageofsucharepresentationresidesinthenumerousoperationsofferedbysuchatransformation.Twotermswhose inferred positions are near one another in this space are equally similar semantically; theserepresentationsaresaidtobedistributed:thevectoroftheconcept‘apartment’[-0.2,0.3,-4.2,5.1...]will be similar to that of ‘house’ [-0.2, 0.3, -4.0, 5.1...]. […] While natural language processing waspioneeringfor‘embedding’wordsinavectorialspace,todaywearewitnessingageneralizationoftheembeddingprocesswhich isprogressivelyextending toallapplications fields:networksarebecomingsimple points in a vectorial space with graph2vec, texts with paragraph2vec, films withmovie2vec,meaningsofwordswithsens2vec,molecularstructureswithmol2vec,etc.AccordingtoYannLeCun,thegoalofthedesignersofconnectionistmachinesistoputtheworldinavector(world2vec).39

Multi-dimensionalvectorspaceisanotherreasonwhythelogicofmachinelearningisdifficulttograsp.Vector space is another new cultural technique, worth becoming familiar with. The field of DigitalHumanities, in particular, has been covering the technique of vectorialisation through which ourcollective knowledge is invisibly rendered and processed. William Gibson’s original definition ofcyberspaceprophesized,mostlikely,thecomingofavectorspaceratherthanvirtualreality:‘Agraphicrepresentationofdataabstractedfromthebanksofeverycomputerinthehumansystem.Unthinkablecomplexity.Linesoflightrangedinthenonspaceofthemind,clustersandconstellationsofdata.Likecitylights,receding.’40

Vectorspaceofsevenwordsinthreecontexts.41

13

Itmust be stressed, however, thatmachine learning still resemblesmore craftsmanship than exactmathematics.AI isstillahistoryofhacksandtricksratherthanmystical intuitions.Forexample,onetrick of information compression is dimensionality reduction, which is used to avoid the Curse ofDimensionality, that is the exponential growth of the variety of features in the vector space. Thedimensionsofthecategoriesthatshowlowvarianceinthevectorspace(i.e.whosevaluesfluctuateonlyalittle)areaggregatedtoreducecalculationcosts.Dimensionalityreductioncanbeusedtoclusterwordmeanings(suchas inthemodelword2vec)butcanalso leadtocategoryreduction,whichcanhave an impact on the representation of social diversity. Dimensionality reduction can shrinktaxonomiesandintroducebias,furthernormalisingworlddiversityandobliteratinguniqueidentities.42

8.Thesocietyofclassificationandpredictionbots.Most of the contemporary applications ofmachine learning can be described according to the twomodalitiesofclassificationandprediction,whichoutlinethecontoursofanewsocietyofcontrolandstatisticalgovernance.Classificationisknownaspatternrecognition,whilepredictioncanbedefinedalsoaspatterngeneration.Anewpatternisrecognisedorgeneratedbyinterrogatingtheinnercoreofthestatisticalmodel.

Machinelearningclassificationisusuallyemployedtorecogniseasign,anobject,orahumanface,andtoassignacorrespondingcategory(label)accordingtotaxonomyorculturalconvention.Aninputfile(e.g.aheadshotcapturedbyasurveillancecamera)isrunthroughthemodeltodeterminewhether it fallswithin itsstatisticaldistributionornot. Ifso, it isassignedthecorrespondingoutputlabel. Since the times of the Perceptron, classification has been the originary application of neuralnetworks:withDeepLearningthistechniqueisfoundubiquitouslyinfacerecognitionclassifiersthataredeployedbypoliceforcesandsmartphonemanufacturersalike.

Machinelearningpredictionisusedtoprojectfuturetrendsandbehavioursaccordingtopastones,thatistocompleteapieceofinformationknowingonlyaportionofit.Inthepredictionmodality,asmallsampleofinputdata(aprimer)isusedtopredictthemissingpartoftheinformationfollowingonceagainthestatisticaldistributionofthemodel(thiscouldbethepartofanumericalgraphorientedtowardthefutureorthemissingpartofanimageoraudiofile).Incidentally,othermodalitiesofmachinelearningexist:thestatisticaldistributionofamodelcanbedynamicallyvisualisedthroughatechniquecalledlatentspaceexplorationand,insomerecentdesignapplications,alsopatternexploration.43

4-dimension 3-dimension 2-dimension vector space

14

Machine learning classification and prediction are becoming ubiquitous techniques thatconstitutenewformsofsurveillanceandgovernance.Someapparatuses,suchasself-drivingvehiclesand industrial robots, can be an integration of both modalities. A self-driving vehicle is trained torecognisedifferentobjectsontheroad(people,cars,obstacles,signs)andpredictfutureactionsbasedondecisionsthatahumandriverhastakeninsimilarcircumstances.Evenifrecognisinganobstacleonaroadseemstobeaneutralgesture(it’snot), identifyingahumanbeingaccordingtocategoriesofgender,raceandclass(andintherecentCOVID-19pandemicassickorimmune),asstateinstitutionsareincreasinglydoing,isthegestureofanewdisciplinaryregime.ThehubrisofautomatedclassificationhascausedtherevivalofreactionaryLombrosiantechniquesthatwerethoughttohavebeenconsignedtohistory,techniquessuchasAutomaticGenderRecognition(AGR),‘asubfieldoffacialrecognitionthataimstoalgorithmicallyidentifythegenderofindividualsfromphotographsorvideos.’44

Recently, the generative modality of machine learning has had a cultural impact: its use in theproductionofvisualartefactshasbeenreceivedbymassmediaastheideathatartificialintelligenceis‘creative’andcanautonomouslymakeart.AnartworkthatissaidtobecreatedbyAIalwayshidesahumanoperator,whohasappliedthegenerativemodalityofaneuralnetworktrainedonaspecificdataset.Inthismodality,theneuralnetworkisrunbackwards(movingfromthesmalleroutputlayertowardthelargerinputlayer)togeneratenewpatternsafterbeingtrainedatclassifyingthem,aprocessthatusuallymoves fromthe larger input layer tothesmalleroutput layer.Thegenerativemodality,however,hassomeusefulapplications:itcanbeusedasasortofrealitychecktorevealwhatthemodelhaslearnt,i.e.toshowhowthemodel‘seestheworld.’Itcanbeappliedtothemodelofaself-drivingcar,forinstance,tocheckhowtheroadscenarioisprojected. A famous way to illustrate how a statistical model ‘sees the world’ is Google DeepDream.DeepDreamisaconvolutionalneuralnetworkbasedonInception(whichistrainedontheImageNetdatasetmentionedabove) thatwasprogrammedbyAlexanderMordvintsev toprojecthallucinatorypatterns.Mordvintsevhadtheideato‘turnthenetworkupsidedown’,thatistoturnaclassifierintoagenerator,usingsomerandomnoiseorgenericlandscapeimagesasinput.45Hediscoveredthat‘neuralnetworksthatweretrainedtodiscriminatebetweendifferentkindsofimageshavequiteabitoftheinformationneededtogenerateimagestoo.’ InDeepDreamfirstexperiments,birdfeathersanddogeyes started to emerge everywhere as dog breeds and bird species are vastly overrepresented inImageNet. Itwasalsodiscoveredthat thecategory ‘dumbbell’was learntwithasurrealhumanarmalwaysattachedtoit.ProofthatmanyothercategoriesofImageNetaremisrepresented.

INPUT

Input

LABEL

Out

put

Model Model

Model

Pattern

Algorithm

Training data

Out

put

PRIMER

Input

PATTERN

ClassificationPattern recognition

PredictionPattern generation

TrainingPattern extraction

0 1

15

The two main modalities of classification and generation can be assembled in furtherarchitecturessuchasintheGenerativeAdversarialNetworks.IntheGANarchitecture,aneuralnetworkwiththeroleofdiscriminator(atraditionalclassifier)hastorecogniseanimageproducedbyaneuralnetwork with the role of generator, in a reinforcement loop that trains the two statistical modelssimultaneously. For some converging properties of their respective statistical models, GANs haveprovedverygoodatgeneratinghighlyrealisticpictures.Thisabilityhaspromptedtheirabuse inthefabricationof‘deepfakes’.46Concerningregimesoftruth,asimilarcontroversialapplicationistheuseofGANstogeneratesyntheticdataincancerresearch,inwhichneuralnetworkstrainedonunbalanceddatasets of cancer tissues have started to hallucinate cancer where there was none.47In this case‘insteadofdiscoveringthings,weareinventingthings',FabianOffertnotices,‘thespaceofdiscoveryisidenticaltothespaceofknowledgethattheGANhasalreadyhad.[…]WhilewethinkthatweareseeingthroughGAN—lookingatsomethingwiththehelpofaGAN—weareactuallyseeingintoaGAN.GANvisionisnotaugmentedreality,it isvirtualreality.GANsdoblurdiscoveryandinvention.’48TheGANsimulationofbraincancerisatragicexampleofAI-drivenscientifichallucination.

JosephPaulCohen,MargauxLuckandSinaHonari.‘DistributionMatchingLossesCanHallucinateFeaturesinMedicalImageTranslation’,2018.Courtesyoftheauthors.

9.Faultsofastatisticalinstrument:theundetectionofthenew. ThenormativepowerofAIinthe21stcenturyhastobescrutinisedintheseepistemicterms:whatdoesitmeantoframecollectiveknowledgeaspatterns,andwhatdoesitmeantodrawvectorspacesandstatisticaldistributionsofsocialbehaviours?AccordingtoFoucault,inearlymodernFrance,statisticalpower was already used to measure social norms, discriminating between normal and abnormalbehaviour. 49 AI easily extends the ‘power of normalisation’ of modern institutions, among othersbureaucracy,medicineandstatistics(originally,thenumericalknowledgepossessedbythestateaboutitspopulation)thatpassesnowintothehandsofAIcorporations.Theinstitutionalnormhasbecomeacomputationalone:theclassificationofthesubject,ofbodiesandbehaviours,seemsnolongertobean affair for public registers, but instead for algorithms anddatacentres.50‘Data-centric rationality’,PaulaDuartehasconcluded,‘shouldbeunderstoodasanexpressionofthecolonialityofpower.’51

A gap, a friction, a conflict, however, always persists betweenAI statisticalmodels and thehumansubjectthatissupposedtobemeasuredandcontrolled.ThislogicalgapbetweenAIstatisticalmodels and society is usually debated as bias. It has been extensively demonstrated how facerecognitionmisrepresentssocialminoritiesandhowblackneighbourhoods,forinstance,arebypassed

Flair Real T1 Transformed T1 Real Flair Real T1 Transformed T1 Real

A translation removing tumors A translation adding tumors

16

byAI-drivenlogisticsanddeliveryservice.52Ifgender,raceandclassdiscriminationsareamplifiedbyAIalgorithms,thisisalsopartofalargerproblemofdiscriminationandnormalisationatthelogicalcoreof machine learning. The logical and political limitation of AI is the technology’s difficulty in therecognition and prediction of a new event. How is machine learning dealing with a truly uniqueanomaly, an uncommon social behaviour, an innovative act of disruption? The two modalities ofmachinelearningdisplayalimitationthatisnotsimplybias.

A logical limit of machine learning classification, or pattern recognition, is the inability torecogniseauniqueanomalythatappearsforthefirsttime,suchasanewmetaphorinpoetry,anewjoke in everyday conversation, or an unusual obstacle (a pedestrian? a plastic bag?) on the roadscenario.Theundetectionofthenew(somethingthathasnever‘beenseen’byamodelandthereforeneverclassifiedbeforeinaknowncategory)isaparticularlyhazardousproblemforself-drivingcarsandonethathasalreadycausedfatalities.Machinelearningprediction,orpatterngeneration,showsimilarfaults in the guessing of future trends andbehaviours. As a techniqueof information compression,machinelearningautomatesthedictatorshipofthepast,ofpasttaxonomiesandbehaviouralpatterns,over the present. This problem can be termed the regeneration of the old— the application of ahomogenousspace-timeviewthatrestrainsthepossibilityofanewhistoricalevent.

Interestingly, inmachine learning, the logicaldefinitionofasecurity issuealsodescribesthelogical limit of its creative potential. The problems characteristic of theprediction of the new arelogically related to those that characterise thegenerationof thenew, because theway amachinelearningalgorithmpredictsatrendonatimechartisidenticaltothewayitgeneratesanewartworkfromlearntpatterns.Thehackneyedquestion‘CanAIbecreative?’shouldbereformulatedintechnicalterms:ismachinelearningabletocreateworksthatarenotimitationsofthepast?Ismachinelearningable to extrapolate beyond the stylistic boundaries of its training data? The ‘creativity’ ofmachinelearning is limited to thedetectionof styles from the trainingdata and then random improvisationwithinthesestyles.Inotherwords,machinelearningcanexploreandimproviseonlywithinthelogicalboundaries that are set by the training data. For all these issues, and its degree of informationcompression,itwouldbemoreaccuratetotermmachinelearningartasstatisticalart.

Another unspoken bug of machine learning is that the statistical correlation between twophenomenaisoftenadoptedtoexplaincausationfromonetotheother.Instatistics,itiscommonlyunderstoodthatcorrelationdoesnotimplycausation,meaningthatastatisticalcoincidencealoneisnot sufficient to demonstrate causation. A tragic example can be found in the work of statisticianFrederickHoffman,whoin1896publisheda330-pagereportforinsurancecompaniestodemonstratea racial correlation betweenbeingablackAmericanandhaving short lifeexpectancy.53Superficiallyminingdata,machinelearningcanconstructanyarbitrarycorrelationthatisthenperceivedasreal.In2008thislogicalfallacywasproudlyembracedbyWireddirectorChrisAndersenwhodeclaredthe‘endoftheory’because ‘thedatadelugemakesthescientificmethodobsolete.’54AccordingtoAndersen,himselfnoexpertonscientificmethodandlogicalinference,statisticalcorrelationisenoughforGoogleto run its ads business, therefore it must also be good enough to automatically discover scientificparadigms. Even Judea Pearl, a pioneer of Bayesian networks, believes that machine learning isobsessedwith ‘curve fitting’, recording correlationswithout providing explanations.55Such a logicalfallacyhasalreadybecomeapoliticalone,ifoneconsidersthatpoliceforcesworldwidehaveadoptedpredictive policing algorithms.56According to Dan McQuillan, when machine learning is applied tosociety in thisway, it turns intoabiopoliticalapparatusofpreemption, thatproduces subjectivitieswhich can subsequentlybe criminalized.57Ultimately,machine learningobsessedwith ‘curve fitting’imposes a statistical culture and replaces the traditional episteme of causation (and politicalaccountability)withoneofcorrelationsblindlydrivenbytheautomationofdecisionmaking.

17

LewisFryRichardson,WeatherPredictionbyNumericalProcess,CambridgeUniversityPress,1922.

10.Adversarialintelligencevs.artificialintelligence.SofarthestatisticaldiffractionsandhallucinationsofmachinelearninghavebeenfollowedstepbystepthroughthemultiplelensesoftheNooscope.Atthispoint,theorientationoftheinstrumenthastobereversed:scientifictheoriesasmuchascomputationaldevicesareinclinedtoconsolidateanabstractperspective—thescientific ‘viewfromnowhere’, that isoften just thepointofviewofpower.Theobsessive study of AI can suck the scholar into an abyss of computation and the illusion that thetechnicalformilluminatesthesocialone.AsPaolaRicaurteremarks:‘Dataextractivismassumesthateverythingisadatasource.’58Howtoemancipateourselvesfromadata-centricviewoftheworld?Itistimetorealisethatitisnotthestatisticalmodelthatconstructsthesubject,butratherthesubjectthatstructuresthestatisticalmodel.InternalistandexternaliststudiesofAIhavetoblur:subjectivitiesmakethe mathematics of control from within, not from without. To second what Guattari once said ofmachines in general, machine intelligence too is constituted of ‘hyper-developed and hyper-concentratedformsofcertainaspectsofhumansubjectivity.’59

Ratherthanstudyingonlyhowtechnologyworks,criticalinquirystudiesalsohowitbreaks,howsubjectsrebelagainstitsnormativecontrolandworkerssabotageitsgears.Inthissense,awaytosoundthelimitsofAIistolookathackingpractices.Hackingisanimportantmethodofknowledgeproduction,acrucialepistemicprobe into theobscurityofAI.60Deep learningsystems for face recognitionhavetriggered,forinstance,formsofcounter-surveillanceactivism.Throughtechniquesoffaceobfuscation,humanshavedecidedtobecomeunintelligibletoartificialintelligence:thatistobecome,themselves,black boxes. The traditional techniques of obfuscation against surveillance immediately acquire a

18

mathematicaldimensionintheageofmachinelearning.Forexample,AIartistandresearcherAdamHarveyhasinventedacamouflagetextilecalledHyperFacethatfoolscomputervisionalgorithmstoseemultiplehumanfaceswherethereisnone.61Harvey'sworkprovokesthequestion:whatconstitutesaface forahumaneye,ontheonehand,andacomputervisionalgorithm,ontheother?TheneuralglitchesofHyperFaceexploitsuchacognitivegapandrevealwhatahumanfacelooksliketoamachine.Thisgapbetweenhumanandmachineperceptionhelpstointroducethegrowingfieldofadversarialattacks.

AdamHarvey,HyperFacepattern,2016.

Adversarialattacksexploitblindspotsandweakregionsinthestatisticalmodelofaneuralnetwork,usuallytofoolaclassifierandmakeitperceivesomethingthatisnotthere.Inobjectrecognition,anadversarialexamplecanbeadoctoredimageofaturtle,whichlooksinnocuoustoahumaneyebutgetsmisclassifiedbyaneuralnetworkasarifle.62Adversarialexamplescanberealisedas3Dobjectsandevenstickersforroadsignsthatcanmisguideself-drivingcars(whichmayreadaspeedlimitof120km/hwhereitisactually50km/h).63Adversarialexamplesaredesignedknowingwhatamachinehasnever seen before. This effect is achieved also by reverse-engineering the statistical model or bypollutingthetrainingdataset.Inthislattersense,thetechniqueofdatapoisoningtargetsthetrainingdatasetand introducesdoctoreddata. Insodoing italters theaccuracyof thestatisticalmodelandcreatesabackdoorthatcanbeeventuallyexploitedbyanadversarialattack.64

19

Adversarialattackseemstopointtoamathematicalvulnerabilitythatiscommontoallmachine

learningmodels: ‘An intriguingaspectofadversarialexamples is thatanexamplegenerated foronemodel is oftenmisclassified by othermodels, evenwhen they have different architectures orweretrainedondisjoint trainingsets.’65Adversarialattacks remindusof thediscrepancybetweenhumanandmachineperceptionandthatthelogicallimitofmachinelearningisalsoapoliticalone.Thelogicalandontologicalboundaryofmachinelearningistheunrulysubjectoranomalouseventthatescapesclassificationandcontrol.Thesubjectofalgorithmiccontrolfiresback.Adversarialattacksareawaytosabotagetheassemblylineofmachinelearningbyinventingavirtualobstaclethatcansetthecontrolapparatusoutofjoint.AnadversarialexampleisthesabotintheageofAI.

11.LabourintheageofAI.Thenaturesofthe‘input’and‘output’ofmachinelearninghavetobeclarified.AItroublesarenotonlyaboutinformationbiasbutalsolabour.AIisnotjustacontrolapparatus,butalsoaproductiveone.Asjustmentioned,aninvisibleworkforceisinvolvedineachstepofitsassemblyline(datasetcomposition,algorithm supervision,model evaluation, etc.). Pipelines of endless tasks innervate from theGlobalNorthintotheGlobalSouth;crowdsourcedplatformsofworkersfromVenezuela,BrazilandItaly,forinstance,arecrucialinordertoteachGermanself-drivingcars‘howtosee.’66Againsttheideaofalienintelligenceatwork,itmustbestressedthatinthewholecomputingprocessofAIthehumanworkerhas never left the loop, or put more accurately, has never left the assembly line. Mary Gray andSiddharthSuricoinedtheterm ‘ghostwork’ for the invisible labour thatmakesAIappearartificiallyautonomous.

Beyondsomebasicdecisions,today’sartificial intelligencecan’tfunctionwithouthumans inthe loop.Whetherit’sdeliveringarelevantnewsfeedorcarryingoutacomplicatedtexted-inpizzaorder,whentheartificial intelligence(AI)tripsuporcan’tfinishthejob,thousandsofbusinessescallonpeopletoquietlycompletetheproject.Thisnewdigitalassemblylineaggregatesthecollectiveinputofdistributedworkers,shipspiecesofprojectsratherthanproducts,andoperatesacrossahostofeconomicsectorsatalltimesofthedayandnight.67

Automationisamyth;becausemachines, includingAI,constantlycallforhumanhelp,someauthorshave suggested replacing ‘automation’with themore accurate termheteromation.68HeteromationmeansthatthefamiliarnarrativeofAIasperpetuummobileispossibleonlythankstoareservearmyofworkers.

Yet there isamoreprofoundway inwhich labourconstitutesAI.The informationsourceofmachinelearning(whateveritsname:inputdata,trainingdataorjustdata)isalwaysarepresentationofhumanskills,activitiesandbehaviours,socialproductionatlarge.Alltrainingdatasetsare,implicitly,adiagramof thedivisionofhuman labour thatAIhas toanalyseandautomate.Datasets for imagerecognition,forinstance,recordthevisuallabourthatdrivers,guards,andsupervisorsusuallyperformduring their tasks. Even scientific datasets relyon scientific labour, experimentplanning, laboratoryorganisation, and analytical observation. The information flow of AI has to be understood as anapparatusdesigned toextract ‘analytical intelligence’ fromthemostdiverse formsof labourand totransfersuchintelligenceintoamachine(obviouslyincluding,withinthedefinitionoflabour,extendedformsofsocial,culturalandscientificproduction).69Inshort,theoriginofmachineintelligenceisthedivisionoflabouranditsmainpurposeistheautomationoflabour.

20

Historiansofcomputationhavealreadystressedtheearlystepsofmachineintelligenceinthe19th century project of mechanizing the division of mental labour, specifically the task of handcalculation.70The enterprise of computation has since thenbeen a combinationof surveillance anddiscipliningoflabour,ofoptimalcalculationofsurplus-value,andplanningofcollectivebehaviours.71Computationwasestablishedbyandstillenforcesa regimeofvisibilityand intelligibility,not justoflogicalreasoning.ThegenealogyofAIasanapparatusofpowerisconfirmedtodaybyitswidespreademployment in technologies of identification and prediction, yet the core anomaly which alwaysremainstobecomputedisthedisorganisationoflabour.

Asatechnologyofautomation,AIwillhaveatremendousimpactonthejobmarket.IfDeepLearninghasa1%errorrateinimagerecognition,forexample,itmeansthatroughly99%ofroutineworkbasedonvisualtasks(e.g.airportsecurity)canbepotentiallyreplaced(legalrestrictionsandtradeunionoppositionpermitting). The impactofAIon labour iswell described (from theperspectiveofworkers, finally) within a paper from the European Trade Union Institute, which highlights ‘sevenessentialdimensionsthatfutureregulationshouldaddressinordertoprotectworkers:1)safeguardingworkerprivacyanddataprotection;2)addressingsurveillance,trackingandmonitoring;3)makingthepurposeofAIalgorithmstransparent;4)ensuringtheexerciseofthe‘righttoexplanation’regardingdecisionsmade by algorithms ormachine learningmodels; 5) preserving the security and safety ofworkers in human-machine interactions; 6) boosting workers’ autonomy in human–machineinteractions;7)enablingworkerstobecomeAIliterate.’72

Ultimately, the Nooscope manifests for a novel Machinery Question in the age of AI. TheMachineryQuestionwasadebatethatsparkedinEnglandduringtheindustrialrevolution,whentheresponsetotheemploymentofmachinesandworkers’subsequenttechnologicalunemploymentwasasocialcampaignformoreeducationaboutmachines,thattooktheformoftheMechanics’InstituteMovement.73TodayanIntelligentMachineryQuestionisneededtodevelopmorecollectiveintelligenceabout‘machineintelligence,’morepubliceducationinsteadof‘learningmachines’andtheirregimeofknowledgeextractivism(which reinforcesoldcolonial routes, justby lookingat thenetworkmapofcrowdsourcingplatformstoday).AlsointheGlobalNorth,thiscolonialrelationshipbetweencorporateAIandtheproductionofknowledgeasacommongoodhastobebroughttothefore.TheNooscope’spurposeistoexposethehiddenroomofthecorporateMechanicalTurkandtoilluminatetheinvisiblelabourofknowledgethatmakesmachineintelligenceappearideologicallyalive.

21

1Ontheautonomyoftechnologysee:LangdonWinner,AutonomousTechnology:Technics-Out-of-ControlasaThemeinPoliticalThought.Cambridge,MA:MITPress,2001.2Fortheextensionofcolonialpowerintotheoperationsoflogistics,algorithmsandfinancesee:SandroMezzadraandBrettNeilson,ThePoliticsofOperations:ExcavatingContemporaryCapitalism.Durham:DukeUniversityPress,2019.OntheepistemiccolonialismofAIsee:MatteoPasquinelli,‘ThreeThousandYearsofAlgorithmicRituals.’e-flux101,2019.3Digitalhumanitiestermasimilartechniquedistantreading,whichhasgraduallyinvolveddataanalyticsandmachinelearninginliteraryandarthistory.See:FrancoMoretti,DistantReading.London:Verso,2013.4GottfriedW.Leibniz,‘PrefacetotheGeneralScience’,1677.In:PhillipWiener(ed.)LeibnizSelections.NewYork:Scribner,1951,23.5ForaconcisehistoryofAIsee:DominiqueCardon,Jean-PhilippeCointetandAntoineMazières,‘NeuronsSpikeBack:TheInventionofInductiveMachinesandtheArtificialIntelligenceControversy.’Réseaux211,2018.6AlexanderCampoloandKateCrawford,‘EnchantedDeterminism:PowerwithoutControlinArtificialIntelligence.’EngagingScience,Technology,andSociety6,2020.7Theuseofthevisualanalogyisalsointendedtorecordthefadingdistinctionbetweenimageandlogic,representationandinference,inthetechnicalcompositionofAI.Thestatisticalmodelsofmachinelearningareoperativerepresentations(inthesenseofHarunFarocki’soperativeimages).8Forasystematicstudyofthelogicallimitationsofmachinelearningsee:MominMailk,‘AHierarchyofLimitationsinMachineLearning.’Arxivpreprint,2020.https://arxiv.org/abs/2002.051939ForamoredetailedlistofAIbiasessee:JohnGuttagandHariniSuresh,‘AFrameworkforUnderstandingUnintendedConsequencesofMachineLearning.’Arxivpreprint,2019.https://arxiv.org/abs/1901.10002Seealso:AramGalstyan,KristinLerman,NinarehMehrabi,FredMorstatterandNripsutaSaxena,‘ASurveyonBiasandFairnessinMachineLearning.’Arxivpreprint,2019.https://arxiv.org/abs/1908.0963510VirginiaEubanks,AutomatingInequality.NewYork:St.Martin’sPress,2018.Seealso:KateCrawford,‘TheTroublewithBias.’Keynotelecture,ConferenceonNeuralInformationProcessingSystems,2017.11RuhaBenjamin,RaceAfterTechnology:AbolitionistToolsfortheNewJimCode.Cambridge,UK:Polity,2019,5.12ComputerscientistsarguethatAIbelongstoasubfieldofsignalprocessing,thatisdatacompression.13MatteoPasquinelli,TheEyeoftheMaster.London:Verso,forthcoming.14ProjectssuchasExplainableArtificialIntelligence,InterpretableDeepLearningandHeatmappingamongothershavedemonstratedthatbreakingintothe‘blackbox’ofmachinelearningispossible.Nevertheless,thefullinterpretabilityandexplicabilityofmachinelearningstatisticalmodelsremainsamyth.See:ZacharayLipton,‘TheMythosofModelInterpretability.’ArXivpreprint,2016.https://arxiv.org/abs/1606.0349015A.Corsani,B.Paulré,C.Vercellone,J.M.Monnier,M.Lazzarato,P.DieuaideandY.Moulier-Boutang,‘LeCapitalismecognitifcommesortiedelacriseducapitalismeindustriel.Unprogrammederecherché’,Paris:LaboratoireIsysMatisse,MaisondesSciencesEconomiques,2004.Seealso:Zuboff,Shoshana,TheAgeofSurveillanceCapitalism:TheFightforaHumanFutureattheNewFrontierofPower.London:ProfileBooks,2019.16LisaGitelman(ed.),RawDataisanOxymoron,Cambridge,MA:MITPress,2013.17Insupervisedlearning.Alsoself-supervisedlearningmaintainsformsofhumanintervention.18Ontaxonomyasaformofknowledgeandpowersee:MichelFoucault,TheOrderofThings.London:Routledge,2005.19SuchasAmazonMechanicalTurk,cynicallytermed‘artificialartificialintelligence’byJeffBezos.See:JasonPontin,‘ArtificialIntelligence,WithHelpfromtheHumans.’TheNewYorkTimes,25March2007.20AlthoughtheconvolutionalarchitecturedatesbacktoYannLeCunn’sworkinthelate1980s,DeepLearningstartswiththispaper:GeoffreyHinton,AlexKrizhevskyandIlyaSutskever,‘ImageNetClassificationwithDeepConvolutionalNeuralNetworks.’CommunicationsoftheACM60(6),2017.21Foranaccessible(yetnotverycritical)accountoftheImageNetdevelopmentsee:MelanieMitchell,ArtificialIntelligence:AGuideforThinkingHumans.London:Penguin,2019.22WordNetis‘alexicaldatabaseofsemanticrelationsbetweenwords’whichwasinitiatedbyGeorgeArmitageatPrincetonUniversityin1985.Itprovidesastricttree-likestructureofdefinitions.23KateCrawfordandTrevorPaglen,‘ExcavatingAI:ThePoliticsofTrainingSetsforMachineLearning.’19September2019.https://excavating.ai24AdamHarveyandJulesLaPlace,Megapixelproject,2019.https://megapixels.cc/aboutAnd:MadhumitaMurgia,‘Who'sUsingYourFace?TheUglyTruthAboutFacialRecognition.’FinancialTimes,19April2019.25TheGDPRdataprivacyregulationthatwaspassedbytheEuropeanParliamentinMay2018is,however,animprovementcomparedtotheregulationthatismissingintheUnitedStates.26FrankRosenblatt,‘ThePerceptron:APerceivingandRecognizingAutomaton.’CornellAeronauticalLaboratoryReport85-460-1,1957.27WarrenMcCullochandWalterPitts,‘HowWeKnowUniversals:ThePerceptionofAuditoryandVisualForms.’TheBulletinofMathematicalBiophysics9(3):1947.28Theparametersofamodelthatarelearntfromdataarecalled‘parameters’,whileparametersthatarenotlearntfromdataandarefixedmanuallyarecalled‘hyperparameters’(Thesedeterminenumberandpropertiesoftheparameters).29Thisvaluecanbealsoapercentagevaluebetween1and0.30https://keras.io/applications(Documentationforindividualmodels).31PaulEdwards,AVastMachine:ComputerModels,ClimateData,andThePoliticsofGlobalWarming.Cambridge,MA:MITPress,2010.32SeethetheCommunityEarthSystemModel(CESM)thathasbeendevelopedbytheNationalCenterforAtmospheric

22

ResearchinBolder,Colorado,since1996.TheCommunityEarthSystemModelisafullycouplednumericalsimulationoftheEarthsystemconsistingofatmospheric,ocean,ice,landsurface,carboncycle,andothercomponents.CESMincludesaclimatemodelprovidingstate-of-the-artsimulationsoftheEarth'spast,present,andfuture.’http://www.cesm.ucar.edu33GeorgeBox,‘RobustnessintheStrategyofScientificModelBuilding.’TechnicalReport#1954,MathematicsResearchCenter,UniversityofWisconsin-Madison,1979.34Post-colonialandpost-structuralistschoolsofanthropologyandethnologyhavestressedthatthereisneverterritoryperse,butalwaysanactofterritorialisation.35Patternrecognitionisoneamongmanyothereconomiesofattention.‘Tolookistolabor’,asJonathanBellerremindsus.JonathanBeller,TheCinematicModeofProduction:AttentionEconomyandtheSocietyoftheSpectacle.Lebanon,NH:UniversityPressofNewEngland,2006,2.36DanMcQuillan,‘ManifestoonAlgorithmicHumanitarianism.’PresentedatthesymposiumReimaginingDigitalHumanitarianism,Goldsmiths,UniversityofLondon,February16,2018.37AsprovenbytheUniversalApproximationTheorem.38AnanyaGanesh,AndrewMcCallumandEmmaStrubell,‘EnergyandPolicyConsiderationsforDeepLearninginNLP.’ArXivpreprint,2019.https://arxiv.org/abs/1906.0224339Cardonetal,‘NeuronsSpikeBack.’40WilliamGibson,Neuromancer.NewYork:AceBooks,1984,69.41Source:corpling.hypotheses.org/49542JamieMorgenstern,SamiraSamadi,MohitSingh,UthaiponTantipongpipatandSantoshVempala,‘ThePriceofFairPCA:OneExtraDimension.’AdvancesinNeuralInformationProcessingSystems31,2018.43Seetheideaofassistedandgenerativecreationin:RoelofPietersandSamimWiniger,‘CreativeAI:OntheDemocratisationandEscalationofCreativity’,2016.http://www.medium.com/@creativeai/creativeai-9d4b2346faf344OsKeyes,‘TheMisgenderingMachines:Trans/HCIImplicationsofAutomaticGenderRecognition.’ProceedingsoftheACMonHuman-ComputerInteraction2(88),November2018.https://doi.org/10.1145/327435745AlexanderMordvintsev,ChristopheOlahandMikeTyka,‘Inceptionism:GoingDeeperintoNeuralNetworks.’GoogleResearchblog,June17,2015.https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html46Deepfakesaresyntheticmedialikevideosinwhichaperson'sfaceisreplacedwithsomeoneelse'sfacialfeatures,oftenforthepurposetoforgefakenews.47JosephPaulCohen,SinaHonariandMargauxLuck,‘DistributionMatchingLossesCanHallucinateFeaturesinMedicalImageTranslation.’InternationalConferenceonMedicalImageComputingandComputer-AssistedIntervention.Cham:Springer,2018.arXiv:1805.0884148FabianOffert,NeuralNetworkCulturespanel,TransmedialefestivalandKIMHfGKarlsruhe.Berlin,1February2020.http://kim.hfg-karlsruhe.de/events/neural-network-cultures49MichelFoucault,Abnormal:LecturesattheCollègedeFrance1974-1975.NewYork:Picador,2004,26.50Oncomputationalnormssee:MatteoPasquinelli,‘ArcanaMathematicaImperii:TheEvolutionofWesternComputationalNorms.’In:MariaHlavajovaetal.(eds),FormerWest.Cambridge,MA:MITPress,2017.51PaolaRicaurte,‘DataEpistemologies,TheColonialityofPower,andResistance.’Television&NewMedia,7March2019.52DavidIngold,andSpencerSoper,‘AmazonDoesn’tConsidertheRaceofitsCustomers.ShouldIt?’,Bloomberg,21April2016.https://www.bloomberg.com/graphics/2016-amazon-same-day53CathyO’Neil,WeaponsofMathDestruction.NewYork:BroadwayBooks,2016,ch9.54ChrisAnderson,‘TheEndofTheory:TheDataDelugeMakestheScientificMethodObsolete.’Wired,23June2008.Foracritiquesee:FulvioMazzocchi,‘CouldBigDataBetheEndofTheoryinScience?AFewRemarksontheEpistemologyofData-DrivenScience.’EMBOReports16(10),2015.55JudeaPearlandDanaMackenzie,TheBookofWhy:TheNewScienceofCauseandEffect.NewYork:BasicBooks,2018.56ExperimentsbytheNewYorkPoliceDepartmentsincethelate1980s.See:Pasquinelli,‘ArcanaMathematicaImperii.’57DanMcQuillan,‘People’sCouncilsforEthicalMachineLearning.’SocialMediaandSociety4(2),2018.58Ricaurte,‘DataEpistemologies.’59FelixGuattari,SchizoanalyticCartographies.London:Continuum,2013,2.60TherelationshipbetweenAIandhackingisnotasantagonisticasitmayappear:itoftenresolvesinaloopofmutuallearning,evaluationandreinforcement.61AdamHarvey,HyperFaceproject,2016.https://ahprojects.com/hyperface62AnishAthalyeetal.,‘SynthesizingRobustAdversarialExamples.’ArXivpreprint,2017.https://arxiv.org/abs/1707.0739763NirMorgulisetal.,‘FoolingaRealCarwithAdversarialTrafficSigns.’ArXivpreprint,2019.https://arxiv.org/abs/1907.0037464Datapoisoningcanalsobeemployedtoprotectprivacybyenteringanonymizedorrandominformationintothedataset.65IanGoodfellowetal.,‘ExplainingandHarnessingAdversarialExamples.’ArXivpreprint,2014.https://arxiv.org/abs/1412.657266FlorianSchmidt,‘CrowdsourcedProductionofAITrainingData:HowHumanWorkersTeachSelf-DrivingCarstoSee.’Düsseldorf:Hans-Böckler-Stiftung,2019.67MaryGrayandSiddharthSuri,GhostWork:HowtoStopSiliconValleyfromBuildingaNewGlobalUnderclass.Boston:EamonDolanBooks,2019.68HamidEkbiaandBonnieNardi,Heteromation,andOtherStoriesofComputingandCapitalism.Cambridge,MA:MITPress,2017.69Fortheideaofanalyticalintelligencesee:LorraineDaston,‘CalculationandtheDivisionofLabour1750–1950.’BulletinoftheGermanHistoricalInstitute62,2018.

23

70SimonSchaffer,‘Babbage’sIntelligence:CalculatingEnginesandtheFactorySystem’,CriticalInquiry21,1994.LorraineDaston,‘Enlightenmentcalculations’.CriticalInquiry21,1994.MatthewL.Jones,ReckoningwithMatter:CalculatingMachines,Innovation,andThinkingaboutThinkingfromPascaltoBabbage.Chicago:UniversityofChicagoPress,2016.71MatteoPasquinelli,‘OntheOriginsofMarx’sGeneralIntellect.’RadicalPhilosophy2.06,2019.72AidaPonce,‘LabourintheAgeofAI:WhyRegulationisNeededtoProtectWorkers.’ETUIResearchPaper-ForesightBrief8,2020.http://dx.doi.org/10.2139/ssrn.354100273MaxineBerg,TheMachineryQuestionandtheMakingofPoliticalEconomy.Cambridge,UK:CambridgeUniversityPress,1980.Infact,eventheEconomisthasrecentlywarnedabout‘thereturnofthemachineryquestion’intheageofAI.See:TomStandage,‘TheReturnoftheMachineryQuestion.’TheEconomist,23June2016.Thanks to JonBeller,KateCrawford,DubravkoČulibrk,ArianaDongus,ClaireGlanois,AdamHarvey,LeonardoImpett, Katherine Jarmul, Arif Kornweitz, Wietske Maas, Dan McQuillan, Fabian Offert, Godofredo Pereira,JohannesPaulRaether,NataschaSadrHaghighian,OliviaSolis,MitchSpeedandtheextendedcommunityaroundKIMHfGKarlsruhefortheirinputsandcomments.