north americai iotskynet-v2

THE RISE OF THE MACHINE - IS SKYNET CLOSER THAN EVER?

JVM Developer DevOps practitionerDeveloper AdvocateRobot BuilderIoT speculator AI explorer @spoole167

Work at IBM’s UK Researchand Development Laboratory

MeansI get to play

with cool stuff

Steve Poole : IBM

How close are we tobuilding Skynet?

And can anyone do it?

“Skynet is a fictional neural net-based conscious group mind and artificial general intelligence (see also superintelligence)

system that features centrally in the Terminator franchise and serves as the franchise's main antagonist.”

https://en.wikipedia.org/wiki/Skynet_(Terminator)

We live in interesting times

Autonomous Robots

Meet ‘Handle’ Another Robot from Boston Dynamics

http://www.bostondynamics.com/

Handle is 6.5 feet tall, can jump 4 feet and travels at speeds of up to 9 mph. and can travel up to 15 miles between charges

Autonomous Cars

Hyundai - The Empty Car Convoy

Autonomous Drones

The test of the world's largest micro-drone swarm in California in October2016 included 103 Perdix micro-dronesmeasuring around six inches launchedfrom three F/A-18 Super Hornet fighterjets,

Large organisations of devices

Bot Nets

500,000+ hijacked internet-connected thingslike cameras, lightbulbs, and thermostatslaunched the largest DDoS attack everagainst a top security blogger

IOT

An integrated end-to-end solution that enablesyour apps to communicate with, control, analyze,and update connected electronics

50 Billion devices connected to the internet by 2020

Wikipedia: “Skynet gained self-awareness after it had spread into millions of computer servers all across the world”

AI’s that can beat humans at Chess, Go and even Jeopardy

Jan 2016Feb1996 Feb 2011

Computers we can talk to

Apple’s Siri Google’s AssistantAmazon’s Alexa

Computers that can understand the world

https://github.com/karpathy/neuraltalk2

Dynamic Image Captioning

Computers that can recognise you

Q: How long do we have left?In the movie (set in 1984) the first Terminator came from 2024

A: AI is not scary in the way you might imagine

The Nature of AI: a worked example

CanIbuildaterminator?

Mywifesays“no”

CanIbuildaterminator?

Toolate!

CanIbuildarobotthatdoessomethingelse?

Naturallymycolleaguessuggestweshouldbuildabeerdeliverysystem

Terminator4J RobotDelivery4J

MyRobot

• Needstobeabletodeliveracantoanindividualwithinaspecificamountoftimeiebeforethedrinkgetstoowarm

• Needstotakeverbalinstructionsnokeyboardsthankyou

• Needstobeabletocommunicatewiththetargetindividualie”hereisyourdrink”

• Plusallthesimplestufflikefindingitswayaroundaroom

MyRobotneeds“senses”

Hear->microphoneSpeak->loudspeakerLocate->position&orientation?Touch->ultra-sonicrangesensor?Vision->cameraTemperature->temperaturesensor?

Mytoolkit

• Java• OpenCV• Cuda4J• ApacheSpark• DeepLearning4j• Neuroph• Processing• Gazebo

• RaspberryPI• RobotHAT• Robotchassiswithwheels• TiSensorTag• Ultrasonicrangefinder• Webcam• Loudspeaker• Microphone• Batteriesxmany,many

MyRobot

DistanceSensor

Wheels

TiTag

Camera

Raspberry PIRobot HATBattery

MyRobot

DistanceSensor

Wheels

TiTag

Camera

Drink

TiSensorTag

• Bluetoothenabled

• 9axismotionsensor• IRthermopiletemperaturesensor• Digitalmicrophone• Magnetsensor• Humiditysensor• Pressuresensor• Lightsensor

Howhardcanitbe?

Somethingsimple– directioncontrol

LearningwithNeuralNets

• Simpleexample

• Iftargettotheleft- turnleft• Iftargettotheright– turnright• Iftargetstraightahead– moveforwards• Iftargetwithin5cm- stop

Target

DistanceSensor

Wheels

0o

180o

LeftMotor=backRightMotor=forward

LeftMotor=forwardRightMotor=back

LeftMotor=forwardRightMotor=forward

LeftMotor=stopRightMotorstop

Psudocode

ifdistancetotarget<=5cm//closeleftmotor=stoprightmotor=stop

elseifangletotarget>355oor<5o //straightaheadleftmotor=forwardrightmotor=forward

else ifangle<180o //ontherightleftmotor=forwardrightmotor=backward

Else//ontheleftleftmotor=backwardrightmotor=forward

Motordirectionplot(distanceignored)

0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345

Left Right

+0.5Forward

-0.5Backwards

Angleoftarget

0.5 0.7

0.52

ValuesfromNodes

Areweighted0.10.9

summed

=(0.5*0.9)+(0.7*0.1)=0.45+0.07=0.52

Andasignmoidtransferfunctionapplied

1.0./(1.0+exp(-0.52))

0.6271477663131956

0.5 0.7

0.52

0.1- 0.010.9+0.02

0.6271477663131956

Wetakeexpectedanswer 0.7

Andadjusttheweights

Dependingontheircontribution

=(0.5*0.92)+(0.7*0.09)=0.46+0.063=0.523

0.6278489986434628Andrepeatafewtimes

Myfirstneuralnet

Inputnode“direction”

Hiddenlayer

Outputmode“rightmotor”

Outputmode“leftmotor”

Myfirstneuralnet

• Allinputandoutputisbetween0&1• Usesa“sigmoidfunction”asthetrigger• Usesabackpropagationmethod• Usessupervisedtraining.

Motordirection50sampledata

-0.07

-0.035

0.

0.035

0.07

0.105

0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345

Left Right

+0.5Forward

-0.5Backwards


-0.75

-0.5

-0.25

0.

0.25

0.5

0.75

0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345

Left Right

+0.5Forwards

-0.5Backwards


-0.75

-0.5

-0.25

0.

0.25

0.5

0.75

0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345

Left Right

-0.5Backwards

+0.5Forwards


-0.8

-0.5

-0.3

0.

0.3

0.5

0.8

0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345

Left Right

-0.5Backwards

+0.5Forwards

But..

ErrorsinyourtrainingcancauseerrorsintheAI

Motordirectionplot(distanceignored)

-0.8

-0.5

-0.3

0.

0.3

0.5

0.8

0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345

Left Right

Ihadabuginthetrainingalgorithm


-0.75

-0.5

-0.25

0.

0.25

0.5

0.75

0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345

Left Right

WhichtheNNcopied

Whataboutotherneuralnetstructures

1x2x2?1x12x2?

Motordirection500000sampledata1x2x2

-0.75

-0.5

-0.25

0.

0.25

0.5

0.75

0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345

Left Right

Motordirection500000sampledata1x12x2

-0.75

-0.5

-0.25

0.

0.25

0.5

0.75

0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345

Left Right

Trainingcosts

• Morenodes->moretime(noguaranteeofincreasedaccuracy)• Moreaccuracy->moretraining->(noguaranteeofincreasedaccuracy)

• Increasedcomplexityofproblem->morenodes—>moretime->(noguaranteeofincreasedaccuracy)

We’veseenasimpleNNlearningtheoutputofanalgorithm

• Thealgorithmisasortofmathematicalfunction

• Italready‘knows’howtomaptheinputvaluetosomeoutputvalues

• TheNNistrainedtofindthesamefunctionmappingina3dimensionalspace.

• Somewhereinthespaceof(angle,rightmotor,leftmotor)thereistherightanswer.

Nowwe’lladdindistance• we’recreatinganewalgorithmthatmapsavalueina4dimensionalspace

• Angle,distance,leftmotor,rightmotor

Distance

Mysecondneuralnet

Inputnode“distance”

Hiddenlayer

Outputmode“rightmotor”

Outputmode“leftmotor”

Inputnode“direction”

AIVisualiserScreenshot

Poorlytrained

Loop 5000 timespick random angleget expected answerget answer from neural netapply positive & negative feedback

Loop throw angles 0..360get expected answerget answer from neural netapply positive & negative feedback

Properlytrained

Nextsteps.

• Addintheothersensors• Temperature,proximitysensor,acceleration,light,compassdirectionetc

Distance

I’mgoingtoneedabiggerneuralnet

left motor right motor I’m stuck No drink drink too warmdrink delivered

?

HowdoIdecideonthestructureofthenetwork?

HowdoItrainthisthing!!!

HowdoIvisualisehowtheAI‘thinks’?

Training=Simulation+Neuroevolution

• Gazeboforrobotsimulation• UsegeneticalgorithmstorepresentNeuralNetwork.• Definefitnessselectionsandaddincrementally• Fastesttodeliver• Nearesttotarget• abletospotbeingstuck…

• Evolve.• Willletyouknowwhenthisworks.• Unsupervisedtraining- survivalofthefittest

Terminator4J RobotDelivery4J

• Trainingisstupidlyhard• Ihadn’tappreciatedhowhard.• YouneedsomethinglikeGazebo

• ButitstilltakessignificanttimetotrainandAI• PlusvisualisingtheAIdecisionprocessisdifficultifnotimpossible

http://gazebosim.org/

Computer Vision : face recognition

• How do you turn a picture of a face into a ‘key’ ?

• Especially when the face is at a different distance or relative angle?

“face landmark estimation"Convert the image into HOG format

Identify key points around the face.

Train an AI to find those points in an image

Use the located points to morph the face image into a standard form

Invented by Vahid Kazemi and Josephine Sullivan.

http://www.csc.kth.se/~vahidk/papers/KazemiCVPR14.pdf

http://sharky93.github.io/docs/gallery/auto_examples/plot_hog.html

Once you can identifythe key parts of a face

you can morph the image in other interesting ways

face landmark estimation

‘Encode” the face • With a standardised face image

• Identify the ‘important’ parts of the face so you can do pattern matching

• The challenge

• We don’t know what the ‘important’ parts are

• The answer

• Use a Neural Network to work it out.

• http://www.cv-foundation.org/openaccess/content_cvpr_2015/app/1A_089.pdf

VoicetoText

• Humansareverygoodatunderstandingspokenwords:-)• Computershaveahardertime

http://cmusphinx.sourceforge.net/

Sphinx4

ApureJavaspeechrecognitionlibrary

WatsonSpeechtoText

JavaAPItoonlinespeechrecognitionsolution(freetier)

https://speech-to-text-demo.mybluemix.net/

Results

“love they do is”

“Open the pod bay doors hal.”

Speaker 0: Open the pod baydoors hal.

Speaker 1: Sorry. Great I can.

“Open the pod bay doors hal. I’m sorry Dave. I can’t do that”

Sphinx4 Watson

SpeechtoWords

• Accuratetranslationofspeechtowordscanbeverygood• It’salmostasaccurateashumans• Itwillsoonbemuchbetter.• Withoutgoodmicrophonesresultscanbemixed!• Behindthescenesthereisstillasignificantamountofcomputepowerbeingused.

Practically

• HowwouldIgetmyrobottounderstandcommandslike

• “takethisdrinktoRobinthecorner”• “takethisdrinktotheladytalkingtomywife”• “givethisdrinktothemanwearingtheHawaiianshirtandblackshorts”

• HowdoIextractmeaning?

ChatBots

User: "Alexa, open History Buff"Alexa: "History Buff. What day do you want events for?"User: "August thirtieth."Alexa: "For August thirtieth, in 2003, [...] . Wanna go deeper in history?"User: "Yes."Alexa: "In 1995, Bosnian war [...] . Wanna go deeper in history?"User: "No."Alexa: "Good bye!"

https://github.com/amzn/alexa-skills-kit-java

ChatBots:alittlebitofacheatGetFirstEventIntent get events for {day}GetFirstEventIntent give me events for {day}GetFirstEventIntent what happened on {day}GetFirstEventIntent what happenedGetFirstEventIntent {day}

GetNextEventIntent yesGetNextEventIntent yupGetNextEventIntent sureGetNextEventIntent yes please

AMAZON.StopIntent noAMAZON.StopIntent nopeAMAZON.StopIntent no thanksAMAZON.StopIntent no thank you

Chatbots

• Mapintenttoactions• WorkoutyourintentusingvariouslevelsofNaturalLanguageParsing

• Hardcodedstringmatching• Templates(likeAlexa)• Trainedsentencemapping(likeWatsonConversationService)

• Respondtoyouappropriately• Don’thaveanyintrinsicunderstandingofyourintent(yet)

Artificial Intelligence

• Relies on ‘good’ data representations

• Relies on appropriate internal data structures

• Relies on the ‘right’ sort of training

• Embeds knowledge inside its data structures

• Very quickly becomes opaque to humans

• Needs significant processing power

This is‘art’ not

science

GPUstotherescue?• GPU’scanmassivelyreduceneuralnettrainingandexecutiontimes.

• TheyareideallysuitedtoparallelprocessingthecalculationsofaNeuralNet.• Youcanuseyourownorbuytimeinthecloud• ForinstanceAmazonoffersan16xNVIDIAK80GPUinstanceinECSat$14/hour• Youget,nominally,x10performanceimprovements

• MultiplebindingsforGPUsformostlanguages

• IBMJVMhasintrinsicsupportforGPU’s.JITcanutiliseGPUsunderthecoversforappropriateoperations.

How long do we have left?In the movie (set in 1984) the first Terminator came from 2024

Checkpoint• Buildingyouowncognitiverobotorserviceishard

• Ittakestime.• Ittakesdata.• Ittakespatience.

•WhatAItechniquesaregivingusistheabilitytominedataeffectivelyandbeabletoteachsystemsto‘understand’thatdata

• It’sclearthatcompaniesarecreatingAI’sthatwillbesignificantassets.• It’salsoclearthatdifferentiatorwillbethequalityandquantityofdatausedtotraintheAI

•Nosignofsentienceyet.

•However…

• WedesigntheAI• WetraintheAI• Ourdesires/agendas/biasescaneasilygetencodedintotheAI

• Howdowe‘trust’AI’s?

TheSkynetideastillhasanachillesheel…

Don’tworryaboutSkynetyet.

WorryabouthowwelearntounderstandandvisualisetheAIswe’vecreated.

TheAchillesheelofSkynet

MyRobotuses6AAbatteries

• Givesitafewhoursofrunning

HowmanyAAbatterieswouldyouneedtorunaTerminator?

• Here’stheHumanequivalent

• NumberofAAbatteriestoplaythepianoforanhour?

• NumberofAAbatteriestowalkforanhour?

• NumberofAAbatteriestostandupandpaintawallforanhour?

91

127

152

Yuckyquestion

Howmuchenergy(innumberofAAbatteries)isstoredin1kgofhumanfat?

1869

Soactuallyhumanbeingsareprettyefficientbatteries?

Maybewe’reinthewrongmovie?

Thank you

any questions?

north americai iotskynet-v2

Software