Google,AmazonPatentFilingsRevealDigitalHomeAssistantPrivacyProblems ExecutiveSummaryAnewstudyrevealsthatAmazonandGooglehavefiledpatentapplicationsforanumberoftechnologiesthatwoulddramaticallyexpandsurveillanceofconsumers’privatelives.Thesepatentapplicationsshowhowtechnologycompaniesusehomedatatodrawdisturbinginferencesabouthouseholds,andhowthecompaniesmightusethatdataforfinancialgain.Thetechnologiesdetailedinthepatentapplicationsinclude:

• Asystemforderivingsentimentsandbehaviorsfromambientspeech,evenwhenauserhasnotaddressedthedevicewithits“wakeword.”In2014,asAmazonwaspreparingtorolloutitsEchosmartspeaker,thecompanypatentedanalgorithmthatcouldlistentohumanspeech,includingphoneconversations,forstatementsofinterestsuchas“Iloveskiing.”1Thealgorithmwouldthenprocessthestatementsintokeywordsthatcouldbeusedtotargetadvertising.ThepatentdescribestransmittingthekeywordstoAmazonserversastext,whichcouldallowthecompanytospyonconversationswhiletechnicallykeepingitspromisetoonlystoreandanalyzeaudiorecordingsthatauserintendstoshare.

• Multiplesystemsforidentifyingspeakersinaconversationandbuildinginterestprofilesforeachone.BothGoogleandAmazonofferuserstheoptionofcreatingacoustic“voiceprofiles”forvoice-activatedsmartdevicesintheirhomes.2Theseprofilescanhelpthedevicestailorservicestothepersonspeaking.PatentsshowthatbothAmazonandGooglecouldalsousevoiceprofilestoassociatebehaviorswithindividualmembersofthehousehold,inordertobettertargetads.3

• Amethodforinferringusers’showeringhabitsandtargetingadvertisingbasedonthat

andotherdata.DozensofpatentapplicationsforGoogle’ssmarthomedevicesdetailscenariosinwhichGooglemaysharedatafromsmarthomedeviceswiththirdparties,includingbusinesses,whocanthenusethedatatomakeinferencesaboutusers’sleeping,cooking,entertainment,andshoweringschedules.4Theseinferences,Googlesays,“mayhelpthird-partiesbenefitconsumersbyprovidingthemwithinteresting

1http://pdfaiw.uspto.gov/.aiw?docid=201403371312https://www.amazon.com/gp/help/customer/display.html?nodeId=201602230,https://support.google.com/googlehome/answer/7323910?hl=en3http://pdfaiw.uspto.gov/.aiw?docid=20140337131,http://pdfaiw.uspto.gov/.aiw?PageNum=0&docid=201602593084http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&u=%2Fnetahtml%2FPTO%2Fsearch-adv.html&r=0&f=S&l=50&d=PG01&OS=+%28AANM%2FGoogle+or+AN%2FGoogle%29+and+SPEC%2Fcooking+and+SPEC%2Fshower+and+SPEC%2Finferences&RS=%28%28%28%28AANM%2FGoogle+OR+AN%2FGoogle%29+AND+SPEC%2Fcooking%29+AND+SPEC%2Fshower%29+AND+SPEC%2Finferences%29&TD=80&Srch1=%28%28%28%28Google.AANM.+OR+Google.AS.%29+AND+cooking.BIS.%29+AND+shower.BIS.%29+AND+inferences.BIS.%29&StartNum=&Refine=Refine+Search&Query=+%28AANM%2FGoogle+or+AN%2FGoogle%29+and+SPEC%2Fcooking+and+SPEC%2Fshower+and+SPEC%2Finferences+and+SPEC%2F%22home+data%22

information,productsandservicesaswellaswithprovidingthemwithtargetedadvertisements.”

• Asystemforrecommendingproductsbasedonfurnishingsobservedbyasmarthome

securitycamera.AparticularlytroublingpatentapplicationdescribeshowGooglecouldusevideofeedsfromsmartdevicestodetermineusers’behaviorsandcharacteristics,including“gender,age,fashion-taste,style,mood,knownlanguages,preferredactivities,andsoforth.”5Thepatentapplicationdescribessmartdevicesthattargetadvertisementsbasedonthetitleofthebookbyauser’sbedside,thepresenceofaguitarorbasketballintheroom,andthefaceofafamousactoronauser’st-shirt.

• Amethodologyfor“inferringchildmischief”usingaudioandmovementsensors.

AnotherGooglepatentdescribesasmarthomesystemthatmonitorstheactivityofeverymemberofahouseholdandreportsbacktoadesignated“policyholder.”6Oneapplicationusesvoicefingerprintingtoidentifythepresenceofchildrenandmotionsensorstodetectactivity,andreportsinferredmischiefwhenchildrenarebothactiveandquiet.Thispatentalsomentionsadvertisingapplications.

• Systemsforinsertingpaidcontentintotheresponsesprovidedbydigitalassistants.

BothAmazonandGooglehavepatentedmethodsforservingadstousersthroughtheirsmartdevices.AnAmazondiscussesasystemof“intelligentsponsorshipbasedonknowledgegeneratedbyreferencetothehumanbeingentityusingthesystem.”AGooglepatentshowshowadvertiserscouldbidtohavethecompany’sdigitalassistantfeaturetheirproductsinitsvoice-basedsearchresults.7

WhileAmazonandGooglemaybefocusedonthecommercialimplicationsoftheirinventions,thesetechnologiesalsohavetroublinglegalandethicalimplications.Smartdeviceswithcamerasandmicrophonesinevitablycapturevideooraudioofpeoplewhodidnotconsenttobeingrecorded.8Thirdpartiescanalsoaccessdatafromsmartdevices;arecentstudyfoundthathackerscouldeasilyaccessinsecurewifi-enabled“smart”toysandtalktochildren.9Lawenforcementhasalsobegintosubpoenadatafromsmartdevicesincriminalinvestigations.10Thesepatentapplicationsshowthepotentiallystaggeringcostofafuturistic“smarthome.”Toanticipateandmeetconsumers’needsinnewways,digitalassistantsmakeincreasingly

5http://pdfaiw.uspto.gov/.aiw?Docid=201602601356http://pdfaiw.uspto.gov/.aiw?docid=201602593087http://pdfaiw.uspto.gov/.aiw?docid=201601808488http://www.americancriminallawreview.com/files/5114/9515/4188/ALEXA_AND_THIRD_PARTIES_REASONABLE_EXPECTATION_OF_PRIVACY_FINAL.pdf9https://www.theguardian.com/technology/2017/nov/14/retailers-urged-to-withdraw-toys-that-allow-hackers-to-talk-to-children10http://www.americancriminallawreview.com/files/5114/9515/4188/ALEXA_AND_THIRD_PARTIES_REASONABLE_EXPECTATION_OF_PRIVACY_FINAL.pdf

invasiveforaysintousers’privatelives.Asusersaccepttheseintrusions,theygiveuptheirpersonaldata,andwithit,theirprivacyandsecurity.BackgroundSmartspeakerswithvoice-activateddigitalassistants,liketheAmazonEchoandGoogleHome,promisetobringapersonalStarTrekcomputerintousers’homes.Thedevicesallowuserstosearchtheweb,playmusic,orderpizza,andmakephonecallsviaavoicecommand.11Whenconnectedwithother“smart”devices,theEchoandGoogleHomecanalsodimthelights,controlthethermostat,turnonthesprinkler,andcheckthebabymonitor.12Whilethesedevicesofferconvenience,theyalsooffertechcompaniesunprecedentedinsightintousers’privatelives.Withthesmaller-sizedEchoDotandtheGoogleHomeMini,AmazonandGooglehavebeguntoforaybeyondthelivingroomandintothebedroom.13There,theycaninferfromyourinteractionswiththedevicewhenyouwakeup–andmaybeevenwhoyouwakeupwith.Bothcompanies’eventualgoalistohaveadeviceineveryroomofyourhome.14ThisstudyaskswhatuserdataAmazonandGooglederivefromthesedevicesandwhatthecompaniesintendtodowiththatdata.Thisresearchdrawsfrompatentapplications,whichreflecttheambitiousthinkingofcompanies’researchanddevelopmentteams.Tobesure,thefactthatacompanyhasappliedtopatentaconceptdoesnotmeanthattheywilleverimplementit.Patentsdo,however,reflectacompany’sambitions.PatentapplicationsfortheGoogleGlassandAmazonKindleseemedoutlandishwhentheywerefiled,buttheyledtotheadventofveryrealproductsfortheapplicants.15Thepatentapplicationsdetailedinthismemoprovideinsightintothesurveillancethatispossibleviasmarthomedevices.AmazonEchoAmazon’sdigitalassistant,theEcho,wasthefirstmajorentryintothehomeassistantmarket.Thevoice-activatedsmartspeakercanmakephonecalls,controlsmarthomeandotherinternet-enableddevices,retrievesearchresults,and–ofcourse–orderproductsfromAmazon.com.16DespiteinitialcriticismaboutitsOrwellianimplications,theEchohasachieved 11http://time.com/4906071/alexa-echo-siri-google-home/12https://www.nytimes.com/2016/08/08/technology/amazon-has-big-plans-for-alexa-running-the-star-trek-home.html13https://www.washingtonpost.com/news/the-switch/wp/2017/10/10/the-google-home-mini-is-the-cheapest-way-to-get-into-the-smart-home/?utm_term=.f2e01a925bad14http://www.slate.com/blogs/future_tense/2017/10/05/google_s_home_speakers_are_engineered_to_lock_you_into_google_s_ad_empire.html15https://techcrunch.com/2013/02/21/new-google-glass-patent-is-the-most-comprehensive-yet-for-googles-face-based-wearable-computer/,https://techcrunch.com/2013/02/21/new-google-glass-patent-is-the-most-comprehensive-yet-for-googles-face-based-wearable-computer/16https://www.amazon.com/dp/B07456BG8N

widespreadcommercialsuccess.17Recentestimatessuggestthatmorethan20millionunitshavebeensold.18TheEchoworksbylisteningfora“wakeword”(usually“Alexa,”thenameofthepersonalassistantsoftwarethatrunsontheEcho).19Oncethewakewordisspoken,theEchotransitionsfromwhatAmazoncallsa“passivelisteningstate”toa“responsivestate.”20Inpassivelisteningstate,theEchocapturesandanalyzesallspeechthathappensnearby,butitdoesnotpermanentlyrecordthisspeechortransmititbacktoitsservers.21OncetheEchodetectsthatsomeonesaiditswakeword,itentersaresponsivestate,inwhichrecordstheaudiothatfollowsandsendsittoAmazon’sserversforprocessing.22ItistherethatAmazon’spowerfulnaturallanguageprocessingalgorithmsdeterminewhethertheuserhasaskedAlexatoorderapizzaordimthelights.AmazonclaimsthatitonlystoresandanalyzesaudiospokenfollowinganEcho’swakeword.23WhenuserssetupanewEcho,theymustclickthroughascreenthatfunctionsasconsenttodatacollection.Amazon’sprivacypolicystatesthatthecompanyretainsacousticvoicesignatures,voicequeries,musicplayedthroughthespeaker,andAlexato-doandshoppinglists.24UserscanvisitapageonAmazon’ssitewherethecompanykeepsacompletehistoryoftheuser’squeries.ThesequeriescanadduptovaluableinformationforAmazon.ThesamesophisticatedalgorithmsthatAmazonusestoparsequeriesfromunstructureddata(likespeech)canalsobeusedforwhatthecompanycalls“intelligentadvertising:”

17https://www.bloomberg.com/features/2016-amazon-echo/18https://www.voicebot.ai/2017/10/27/bezos-says-20-million-amazon-alexa-devices-sold/19http://pdfaiw.uspto.gov/.aiw?docid=2017009086420http://pdfaiw.uspto.gov/.aiw?docid=2017009086421https://www.wired.com/2016/12/alexa-and-google-record-your-voice/22https://www.wired.com/2016/12/alexa-and-google-record-your-voice/23https://www.wired.com/2016/12/alexa-and-google-record-your-voice/24https://www.amazon.com/gp/help/customer/display.html?nodeId=201602230,https://www.amazon.com/gp/help/customer/display.html?nodeId=201809740

“Anadditionalwaythatmorerelevantadvertisingcanbedeliveredonsystemspoweredbyembodimentsofthepresentinventionisviaintelligentsponsorshipbasedonknowledgegeneratedbyreferencetothehumanbeingentityusingthesystem.[…]Forexample,auserwithayoungchildwhosebirthdaywascomingupinthenextfewdayscouldbepresentedwithahighly-tailoredintelligentmessagedirectinghimtoalocalbakerywhichsellsbirthdaycakes.”25

AlthoughAmazonclaimsthatitonlysavesaudioofspeechimmediatelyfollowingtheEcho’swakeword,a2014patentapplicationsuggeststhatitcouldalsologalistofkeywordsspokenwhiletheEchoisinapassivelisteningstate.Thepatentapplicationfor“KeywordDeterminationsfromVoiceData”describesasystemthatlistensfornotjustforwakewordsbutalsoforalistofwordsthatindicatestatementsofpreference.26Algorithmsdescribedinthepatenttranslatesthefollowingstatementsintokeywords,andtransmitskeywordsbacktoaremotedatacenter.27Byonlytransmittingkeywordsstrippedofcontext,AmazoncouldcollectmarketingdatafromtheEchowhileitisinpassivelisteningmodewithoutbreakingitspromisetoonlycollectandstoreaudiofollowingthedevice’swakeword.

25http://pdfaiw.uspto.gov/.aiw?docid=2014035128126http://pdfaiw.uspto.gov/.aiw?docid=2014033713127http://pdfaiw.uspto.gov/.aiw?docid=20140337131

Thepatentdescribesa“snifferalgorithm”thatlistensfor“triggerwords.”Inthepatentapplication,Amazondescribestriggerwordsas“averbindicatingsomelevelofdesireorinterestinanounthatfollowsthetriggerwordinasentence.”28Triggerwordscanbepositive(“love,”“enjoy,”“bought,”etc.)ornegative(“hate,”“dislike,“returned,”etc.).Oncethesnifferalgorithmdetectsthatatriggerwordhasbeenspoken,thedevicecanrecordthetriggerwordandtheaudiothatfollows.Unlikewakewords,whichprompttheEchotoimmediatelysendrecordedaudiotoAmazon’sserverforprocessing,29triggerwordscanpromptthesystemtoprocessthephrasefollowingatriggerwordlocallyonthedevice.30Thepatentdescribesan“audioprocessingalgorithm”thatparsesthecapturedspeech.The

algorithmtranslatestheaudiototext,analyzes“closewords”forcontext,removeswordsthatarenotinformative,andassignsapositiveornegativevaluetothekeyword.Theaudioprocessingalgorithmcanalsoassociatekeywordswithspecificindividuals:

“Inembodimentswhereprocessescanattempttodeterminekeywordsformultipleusers,andwheredatabeforetriggerwordsareanalyzedaswell,aprocessmightalsoidentifytheword"I"asanindicatoroftheuserthatshouldbeassociatedwiththatkeyword.Forexample,ifthesentencehadinsteadbeen"Jennylovestoski"thenthatprocessmightassociatethekeyword"ski"withuserJenny(ifknown,identified,etc.)insteadoftheuserspeakingthatsentence.”31

28http://pdfaiw.uspto.gov/.aiw?docid=2014033713129http://pdfaiw.uspto.gov/.aiw?Docid=2017023651230http://pdfaiw.uspto.gov/.aiw?docid=2014033713131http://pdfaiw.uspto.gov/.aiw?docid=20140337131

IllustrationfromPatentApplication14/447487,“KeywordDeterminationsfromVoiceData”

Intheexamplesabove,theaudioprocessingalgorithmidentifiesthesubjectofasentencespokenbyasingleindividual.Thepatentmakesclearthatitcouldalsoparseaconversationbetweentwospeakersandassociatekeywordswitheach.IfapersonconductsaphonecallthroughherEchooranotherdeviceusingthistechnology,theaudioprocessingalgorithmcouldanalyzebothsidesofthecallandrecordkeywordsforbothusers.Whilethepatentforkeyworddeterminationsallowsforthepossibilityofsendingaudiobacktoremoteservers,themostthoroughlydiscussedimplementationsofthetechnologywouldonlytransmitkeywords,sentiment,andtheirassociatedusersbacktoremoteservers.Thedevicecouldtransmitkeyworddatainnearreal-time,atperiodicintervals,orwhenthedevicetransfersdataforanotherpurpose.Thepatentalsonotesthatthedevicecouldtransmitlocationdataalongsidekeywordsinordertofurthertailorproductrecommendations.Amazondoesnotobscurethefactthatthepurposeofthistechnologyisadvertisingandproductrecommendations:

“Oncekeyworddataisstoredforauser,thatkeyworddatacanbeusedtodetermineand/ortargetcontentthatmightbeofinteresttothatuser.[…]Inthisexample,theadvertisingentitylocatedthekeyword"wine"associatedwiththeuserand,basedonanyappropriatecriteriaknownorusedforsuchpurposes,selectedanadrelatingtowinetodisplaytotheuser.Similarly,aproviderofanelectronicmarketplacewhichtheuserisaccessinghasselectedanumberofdifferentproductrecommendations.”

IllustrationfromPatentApplication14/447487,“KeywordDeterminationsfromVoiceData”

Iftheadvertiseroronlinemarketplace(Amazon)hasaccesstoinformationaboutauser’sfriends,itcanusethekeyworddatathatithasderivedfromtwo-wayconversationstomarkettopeoplewhodonotowndevicesequippedwithkeyworddeterminationtechnology.IfyouknowsomeonewithanAmazonEcho,theymaybeabletouseyourconversationswiththatpersontotailoradstoyou.Amazonhasalsodevelopedtechnologytoidentifydifferentusersinonehome.Anotherpatentapplicationdescribesusingvoicesignaturesandbehaviortodistinguishbetweenmembersofahousehold.32Theapplicationdescribesa“masterassociationtable”storedonAmazon’sserversthat“mayincludeinformationassociatedwithindividualusers,households,accounts,etc.thatinteractwiththesystem.”33ThiscouldhelpAmazondeterminewhethertoadvertisebirthdaycaketoyourspouseortoyoursix-year-old.GoogleHomeandAssociatedProductsGoogledebuteditsEchocompetitor,theGoogleHome,in2016.34Asthenamesuggests,Googleintendsforthesmartspeakertoactasahubforavarietyofothersmarthomeproducts.35TheGoogleHomecurrentlyworkswithGoogle’sNestlineofsmartthermostats,cameras,andhazardalarms,aswellasproductsfromSamsung’sSmartThingslineandPhillips’smartlightbulbs.36Google’spatentfilinghistoryshowsthattheGoogleHomeisasteptowardthecompany’seventualgoalofcollectingdatafromeveryroomofthehome.

32http://pdfaiw.uspto.gov/.aiw?docid=2017023651233http://pdfaiw.uspto.gov/.aiw?docid=2017023651234https://www.nytimes.com/2016/11/04/technology/personaltech/google-home-vs-amazon-echo-a-face-off-of-smart-speakers.html35https://www.nytimes.com/2016/11/04/technology/personaltech/google-home-vs-amazon-echo-a-face-off-of-smart-speakers.html36https://www.nytimes.com/2016/11/04/technology/personaltech/google-home-vs-amazon-echo-a-face-off-of-smart-speakers.html

IllustrationfromPatentApplication14/639750,“Smart-HomeAutomationSystemthatSuggestsorAutomaticallyImplementsSelectedHouseholdPoliciesBasedonSensedObservations.

TheGoogleHomeworksinmuchthesamewayastheAmazonEcho.Thevoice-activatedspeakerisalwayslisteningforitswakeword(whichGooglecallsa“hotword”),butitallegedlyonlyrecordsandtransmitsvoicedatafollowinganutteranceofthatword.37However,arecentlydisclosedbugintheGoogleHomeMinicausedanumberofdevicestorandomlyactivateandrecordconversationsevenwhenusersdidnotintendtoactivatethedevice.38Googleclaimsthatithasfixedthebug,buttheincidentshowsjusthoweasyitistotransmitfarmoredatathanintended.LiketheEcho,theGoogleHomeisequippedwithaproprietarydigitalassistantsoftware.Google’sdigitalassistantiscalled“GoogleAssistant”andusersactivateitbysaying,“OkGoogle.39LikeAlexa,Googleassistantcanorderpizzaandperformwebsearches,butitalsoperformsanarrayoffunctionsbasedon“knowledge”drawnfromusers’activityonotherGoogleplatforms.40

37https://www.wired.com/2016/12/alexa-and-google-record-your-voice/38https://www.theverge.com/2017/10/10/16456050/google-home-mini-always-recording-bug39https://assistant.google.com/40https://www.theverge.com/2017/5/17/15651246/google-assistant-iphone-ai-future-interface-io-2017

TheGoogleAssistantmakesinferencesbasedusers’webbrowsinghistories,emailcontents,andlocationdatafromGoogle’smobileapps.41ApatentapplicationforGoogle’svoicerecognitionsystemsaysthatitdrawsfrom“dataincludingauser'sgeographiclocation,auser'ssearchhistory,user'sinterests,orauser'sactivity”todeterminetheappropriateanswertoaspokenquery.42Googleusespastvoicequeriestotargetadsandtrainitssystemtoprovidemoretailoredanswersinthefuture.43WhenuserssetupanewGoogleHome,theymustclickabuttontoacceptGoogle’sTermsofServiceandPrivacyPolicy.ThosedocumentsgiveGooglebroadlicensetouseanydatathatpassesthroughitssystemtoprofileandtargetadstousers.44Theset-upscreenforNestdevicescontainsasimilarscreenthatwillnotletusersproceeduntiltheyacceptthecompany’sTermsofServiceandPrivacyStatement.Nest,anAlphabetsubsidiary,hasaseparatePrivacyStatementthatpurportstograntusersmorecontrolofthedatathatthecompanycollectsandshareswithothers.45ThePrivacyStatementdoesnot,however,indicatewhetherGoogleconstitutesathirdpartyforinformationsharingpurposes,nordoesitclarifywhetherthepolicyalsoappliestointernalprocessingofuserdataforadtargetingpurposes.46TheGoogleHomeandconnectedsmarthomedevicesallowGoogletogathernewtypesofinformationaboutitsusers.LikeAmazon,Googlehasfiledpatentapplicationsforitsmethodofidentifyingandprofilingdifferentspeakersbasedoncapturedaudio.47Googlecansupplementtheseprofileswithdatathatitgathersfromthesensorsandmicrophonesonitssmarthomeproducts.Theseproductscancapturenoisesignatures,moisturelevels,subtletemperaturechanges,lightlevels,andotherdatathatindicatewhatishappeninginsidethehome.DozensofpatentapplicationsforGoogleproductscheerfullyannouncethatthecompany’ssmarthomedevicesmaybeusedtodraw“inferences”abouttheirusers:

41https://www.theverge.com/2017/5/17/15651246/google-assistant-iphone-ai-future-interface-io-201742http://pdfaiw.uspto.gov/.aiw?docid=2017019399943https://www.google.com/intl/en-US_us/policies/technologies/pattern-recognition/,https://support.google.com/assistant/answer/7126196?co=GENIE.Platform%3DAndroid&hl=en44https://www.google.com/policies/privacy/45https://nest.com/legal/privacy-statement-for-nest-products-and-services/46https://nest.com/legal/privacy-statement-for-nest-products-and-services/47http://pdfaiw.uspto.gov/.aiw?docid=20160293167,http://pdfaiw.uspto.gov/.aiw?docid=20170110144

“[S]uchinferencesmayincludewhenareoccupantshome,whenaretheysleeping,whenaretheycooking,whenaretheyinthedenwatchingtelevision,andwhendotheyshower.Theanswerstothesequestionsmayhelpthird-partiesbenefitconsumersbyprovidingthemwithinterestinginformation,productsandservicesaswellaswithprovidingthemwithtargetedadvertisements.”48

ManyotherpatentsforGoogle’ssmarthomeproductscontainboilerplatedescribingan“advertising/communicationparadigm[…]thatestimatescharacteristics(e.g.,demographicinformation),desiresand/orproductsofinterestofauserbasedondeviceusage.”49Onepatentapplication,ironicallynamed“Privacy-AwarePersonalizedContentfortheSmartHome,”goesevenfurther:

“Insomeembodiments,therobot69mayuseitssensorstomeasuresoundsandappearanceofpeopleand/orobjectsintheroomandsendaudioand/orvisualfeaturesthatcanbeusedtoassesspropertiesoftheobjectsand/orpeople.Forexample,forpeople,theaudioand/orvisualfeaturesmaybeusedtoascertaingender,age,fashion-taste,style,mood,knownlanguages,preferredactivities,andsoforth.”50

Theapplicationgoesontodescribemarketingsportscamptoa15-year-oldboyholdingabasketballinthelivingroom,combiningbrowsersearchhistorywithanimageonauser’st-shirttoinferaninterestintheactorWillSmith,andrecommendingaTVprogrambasedonthebookonauser’sbedsidetable.51

48Seee.g.http://pdfaiw.uspto.gov/.aiw?Docid=2016026013549Seee.g.http://pdfaiw.uspto.gov/.aiw?Docid=2017030096550http://pdfaiw.uspto.gov/.aiw?Docid=2016026013551http://pdfaiw.uspto.gov/.aiw?Docid=20160260135

IllustrationfromPatentApplication14/638937,“Privacy-AwarePersonalizedContentfortheSmartHome.”

IfthatwerenotOrwellianenough,anotherpatentapplicationdescribesusingasmarthomesystemtomonitorandcontrolscreentime,hygienehabits,mealandtravelschedules,andotheractivity.Thesystemevenpurportsto“infermischief”basedonaudioandmotionsensorreadingsfromroomswherechildrenarepresent.52Eventhispatentapplication,whichdescribesasystemwithnear-totalinsightintoeveryactivityinusers’homes,containsaprovisothatdeviceusagedatamaybeusedforadvertising.

52http://pdfaiw.uspto.gov/.aiw?docid=20160259308

Google’ssmarthomepatentapplicationsalsodescribethecompany’sinterestinusers’healthdata.Severalapplicationsdescribecreatinganaudiosignatureforthemasterbathroomthatwillrecordthenumberoftimesthatatoiletisflushed.53Anotherapplicationdescribesmonitoringauser’snighttimebreathingpatternstodetectasleepdisorder,andstoringtheinformationinauser’s“sleepprofile.”54Whilethesepatentapplicationsdonotdescribesystemsforadvertisingtousersbasedonthisinformation,Googlehasdemonstratedawillingnesstoadvertisetousersbasedonhealthdataderivedfromotherproducts.Forexample,a2015patentdescribesaschemetoadvertisevacationstoindividualswhoseheartratesindicatethattheyareunderstress.55 Google’svastrepositoryofinformationaboutitsusershasmadethecompanythesingle-biggestrecipientofdigitaladvertisingrevenues.56Truetoform,Googlehasappliedforapatenttobringitsubiquitousdisplayadauctionmodeltoitsvoice-controlledassistant.57ThepatentdescribesasystembywhichadvertiserscanbidtohavetheircustomizedvoiceactionsinsertedintoresponsestoGoogleHomeusers’queries:

“Inoneexample,thevoiceinputof"findrestaurantnearMountainView,Calif."maycauseLittleItalytoappearatthetopofthelistofrestaurantspresentedonuserdevice104.Inanotherexample,thevoiceinputof"findrestaurantnearMountainView,Calif."maycauseamessagetobepresentedtotheusersuggestingthattheusertryLittleItalybeforethelistofrestaurantsispresented.“58

53http://pdfaiw.uspto.gov/.aiw?Docid=2017026311154http://pdfaiw.uspto.gov/.aiw?Docid=2017012798055http://pdfaiw.uspto.gov/.aiw?Docid=2017011244556https://www.emarketer.com/Article/Microsoft-Surpass-Yahoo-Global-Digital-Ad-Market-Share-This-Year/101101257http://pdfaiw.uspto.gov/.aiw?docid=2016018084858http://pdfaiw.uspto.gov/.aiw?docid=20160180848

IllustrationfromPatentApplication14/639750,“Smart-HomeAutomationSystemthatSuggestsorAutomaticallyImplementsSelectedHouseholdPoliciesBasedonSensedObservations.

ThismakestheHomeamoneymakingprospectforGoogleintworespects:Itnotonlygathersdatatoservemoretargetedadsonusers’phonesandcomputers,butitalsocreatesanotherplatformonwhichtoservepaidcontent.ImplicationsThedigitaladvertisingofthelasttwodecadestargetedadsprimarilyonthebasisofdemographicinformationandinterestssuppliedbytheuserthroughsearchqueries,emailcontents,andaccountinformation.Moderndigitaladvertisingfocusesoninferredbehaviors,contexts,andemotionsasbetterindicatorsofwhatconsumersarelikelytobuy.59Whenconsumersareattheircomputers,companieslikeGoogleandAmazonobservetheirbehaviorandemotionsthroughbrowsercookiesthattrackthewebsitesthattheyvisit.60Whenconsumersareontheirmobilephones,GoogleandAmazoncanobservetheirbehaviorandsurroundingsthroughlocationdata.61DigitalassistantsandsmarthomedevicesallowcompanieslikeGoogleandAmazonaccesstotheoneplacewhereusersarelikelytospendtimedisconnectedfromtheirdevices:home.Thesepatentsshowthatsmartdevicestargetmomentsinbetweenscreentimetomonitorsleephabits,listeninondinnerconversations,andtrackwhenusersshower.AccesstothisdatacanfleshoutGoogleandAmazon’sprofilesoftheirusersinordertohelpthemmoreaccuratelyservertargetedads.Smartdevicesalsoofferanewadvertisingplatforminthemselves–a2013letterfromGoogletotheSecuritiesandExchangeCommissionpredictedthatinafewyears,thecompanycouldserveadstorefrigerators,thermostats,andothereverydaydevices.62Whiletechcompaniesareprimarilyinterestedincommercialusesforhomedata,creatingarepositoryofinformationaboutusers’privatelivesraisesthenear-certaintythatotherswillattempttoaccessit.HomeinsurersandutilitycompanieshavealreadymadedealswithNesttoputsmartdevicesintheircustomers’homes.63Otherbusinessesthatwouldbenefitfromhomedata,likelendersandhealthinsurers,couldbenext.Lawenforcementhasalreadyfoundauseforsmartdevices.AnAmazonEchomadeheadlineslastyearwhenpoliceinvestigatingamurdersoughttosubpoenarecordingsmadebythedevice.64Investigatorsinthesamecasealsomanagedtoobtaindatafromasmartwatermeter

59http://adage.com/article/digitalnext/bem-a-approach-audience-targeting/308383/60https://www.amazon.com/gp/help/customer/display.html?nodeId=202075050,https://www.google.com/policies/technologies/types/61https://www.google.com/policies/privacy/example/may-collect-and-process-information.html,https://www.amazon.com/gp/help/customer/display.html/ref=hp_terms_us?nodeId=20160420062https://www.sec.gov/Archives/edgar/data/1288776/000128877613000074/filename1.htm63https://nest.com/insurance-partners/,https://nest.com/energy-partners/64http://www.americancriminallawreview.com/files/5114/9515/4188/ALEXA_AND_THIRD_PARTIES_REASONABLE_EXPECTATION_OF_PRIVACY_FINAL.pdf

thatsuggestedthatthecrimescenehadbeenhoseddownbeforepolicearrived.65In2015,PopularMechanicsreportedthatpolicehadrequesteddatafromNestcamerasonmultipleoccasions,thoughitisunclearwhetherGooglehascompliedwiththerequests.66Hackersorothermaliciousactorscouldalsoaccesstouserdatathroughtheirsmartdevices.GoogleHome’sFAQcontainsthefollowingdisclaimer:

“AnyonewhoisnearyourGoogleHomedevicecanrequestinformationfromit,andifyouhavegivenGoogleHomeaccesstoyourcalendars,Gmailorotherpersonalinformation,peoplecanaskyourGoogleHomedeviceaboutthatinformation.GoogleHomealsogetsinformationaboutyoufromyourotherinteractionswithGoogleservices.”67

Arecently-disclosedBluetoothvulnerabilitycouldallowhackerstocompletelytakeoverboththeEchoandtheGoogleHome,allowingaccesstoallprivateinformationthatpassesthroughthedevice.68Anotherrecentstudyshowedthatsomesmartdevicesaresoinsecurethathackerscanusethemlikeintercomstotalktochildren.69Digitalassistantsandothersmartdevicesalsoraisethornyissuessurroundingtheconstitutionally-guaranteedreasonableexpectationofprivacy.GueststoEcho-orGoogleHome-enabledhomesmaynotnecessarilyconsenttobeingrecorded.Legalscholarsarguethatownersofasmarthomedeviceshouldhaveadutytodisclosethepresenceofthedevicetoguests.70Nest’stermsofserviceexplicitlyplaceliabilityforfailuretoobtainconsentbeforerecordingontheusersofitsdevices.71Inpractice,however,smartdeviceownersareunlikelytoexplicitlywarntheirdinnerguestsortrick-or-treatersthattheymightberecorded.DigitalassistantsliketheAmazonEchoandtheGoogleHomegreatlyexpandthecollectionofpersonaldata,magnifyingtheriskthatsomeonewilllearnsomethingaboutyouthatyouwouldratherkeepprivate.

65http://www.americancriminallawreview.com/files/5114/9515/4188/ALEXA_AND_THIRD_PARTIES_REASONABLE_EXPECTATION_OF_PRIVACY_FINAL.pdf66http://www.popularmechanics.com/technology/security/a14171/dropcam-law-enforcement-video-requests/67https://support.google.com/googlehome/answer/7072285?hl=en68https://www.armis.com/blueborne-cyber-threat-impacts-amazon-echo-google-home/69https://www.theguardian.com/technology/2017/nov/14/retailers-urged-to-withdraw-toys-that-allow-hackers-to-talk-to-children70http://www.americancriminallawreview.com/files/5114/9515/4188/ALEXA_AND_THIRD_PARTIES_REASONABLE_EXPECTATION_OF_PRIVACY_FINAL.pdf71 https://nest.com/legal/terms-of-service/