Download - TouchNet preview at Numenta
![Page 1: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/1.jpg)
TouchNet preview
by Jason Toy
![Page 2: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/2.jpg)
TouchNet
• A touch simulator and dataset of touchable objects to teach AIs how to interact with their environments via motor-sensory neurons
![Page 3: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/3.jpg)
history
• idea around 2015
• started coding September 2017
• initial release 2017 Q4
![Page 4: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/4.jpg)
why TouchNet?
![Page 5: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/5.jpg)
why TouchNet?
• On Intelligence
• future predicting machines
• neuroscience based principles
![Page 6: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/6.jpg)
why TouchNet?
• Helen Keller
• intelligence through fingers
• consciousness “started” from I
![Page 7: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/7.jpg)
Helen Keller• “Every object is associated in my mind with tactual
qualities which, combined in countless ways, give me a sense of power, of beauty.”
• “..with the fingers I see only a very small portion of a surface, and that I must pass my hand continually over it before my touch grasps the whole. It is still more important, however, to remember that my imagination is not tethered to certain points, locations, and distances. It puts all the parts together simultaneously as if it saw or knew instead of feeling them.”
![Page 8: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/8.jpg)
where is AI now?• images, CNNs, LSTMs, GANs
• most business focus on static models; some work in RL
• where is time?
• motor sensory neurons?
• where is the agent compared to the world?
![Page 9: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/9.jpg)
- John Locke, philosophy of mind
“An unknown, but potentially large, fraction of animal and human intelligence is a direct
consequence of the perceptual and physical richness of our environment, and is unlikely to
arise without it.”
![Page 10: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/10.jpg)
TouchNet consists of 4 parts
• dataset of 3D objects to touch and interact with
• an evolving simulation
• a simple extendable API
• benchmarks to measure and track progress
![Page 11: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/11.jpg)
Dataset
![Page 12: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/12.jpg)
Dataset
• procedurally generated 3D objects
• common objects: cups, plates, bowls, etc
• starting with tool objects
• cifar100 …. imagenet
![Page 13: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/13.jpg)
Simulator
![Page 14: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/14.jpg)
Simulators and physics
• need a real physics engine
• needs to be fully programmable & modifiable
• tested all of them: MuJoCo, DeepMind Lab, Roboschool, project Malmo, VizDoom,Gazebo, bullet , Processing
![Page 15: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/15.jpg)
Simulators and physics• bullet and pybullet are a secret weapon!
• robust physics engine used for many applications
• super easy to install
• movies: Shrek, Hancock,Sherlock Holmes….
• games: Toy Story 3, Grand Theft Auto 4,Hot Wheels…
![Page 16: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/16.jpg)
pybullet• rigid and soft body simulations
• collision detection
• works with unity, blender,CUDA,OpenCL,…..
• developed by Erwin Coumans & community since ~2003
• now at Google and used for RL & Robotics
• http://pybullet.org
![Page 17: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/17.jpg)
What we want to simulate
![Page 18: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/18.jpg)
What we have
![Page 19: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/19.jpg)
simulation demo
![Page 20: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/20.jpg)
coevolving simulator & agent
• index finger with merkel touch sensors
• all sensor types in finger
• all fingers
• full arm
• full body?
![Page 21: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/21.jpg)
API
• simple to get started and use
• easy to install
• modifiable and customizable
![Page 22: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/22.jpg)
reinforcement learning api
• based off OpenAI’s gym and universe API
• can plug in almost any code for reinforcement learning
![Page 23: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/23.jpg)
OpenAI’s Universe API
![Page 24: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/24.jpg)
Same API
![Page 25: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/25.jpg)
Benchmarks
![Page 26: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/26.jpg)
“Blind” objection classification
• goal: classify the object
• CNN and actor critic DNN models trained concurrently
• pytorch and tensorflow versions
• 4% accuracy, please beat mine!
![Page 27: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/27.jpg)
“blind” classification code walk through
![Page 28: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/28.jpg)
Benchmarks
• HTM version?
• more benchmarks in the future e.g:
• moving objects, differentiating textures,…..
![Page 29: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/29.jpg)
Help!
• very ambitious project, too big for me
• 100% open source
• I do a little of everything ( suck at everything? )
• looking for contributors and experts in physics,math, neuroscience, ml
![Page 30: TouchNet preview at Numenta](https://reader036.vdocuments.us/reader036/viewer/2022062523/5a649b6d7f8b9a2c568b638f/html5/thumbnails/30.jpg)
Jason Toy [email protected]
twitter.com/jtoy
Thank you!
Questions and Feedback
http://github.com/jtoy/touchnet