"imaging + ai: opportunities inside the car and beyond," a presentation from lux research
TRANSCRIPT
Imaging + AI: Opportunities Inside the Car and Beyond Embedded Vision Alliance Member Meeting
Mark Bünger, VP of Research
Lux Research www.luxresearchinc.com
Autonomous Systems 2.0 Sensors Electronic User Interfaces
December 2016
Emerging technologies’ impact on the material world… is generally not pretty
“I’ll call you on my camera.” -no one, ever
Digital transformation means industry transformation
Aka disruption, paradigm shift, gales of creative destruction
One of the most common patterns of transformation is that adjacent industries merge
And only one of the two survives
2
Finding high-potential vehicle vision technologies
3 Technology Readiness
Te
chn
olo
gy
Att
ract
ive
ne
ss
Scale Deployment Concept Stage
Mo
st a
ttra
ctiv
e L
east
att
ract
ive
Which sensors are best for these applications?
Specialized, precision sensor
Arrays of simple sensors (e.g. image, mic) combined with inexpensive AI
4
“Sensor substitution” or “software-defined sensors” - increasingly the right answer Why?
• Software innovation cycles faster than hardware • Compelling economics • Strong historical precedents • Ecosystem
OR
Imaging and audio sensors spread everywhere, to everyone
Billions of smartphone cameras
Lifeloggers shrinking in size and entering consumer market
Memoto Narrative Clip
Autographer
ActionCam (GoPro) market growing ~20% annually
Russian dashboard cameras are awesome
Google launches Glass, kind of a flop, but probably not last try
Police-mounted cameras
5
See “Seeing the Value in Machine Vision Partnerships” Nov 2014
Imaging and audio sensors + AI = sensor substitution, anything is an interface. Big Deal
Depth Camera visual interfaces
Microsoft Kinect recognizes body and facial gestures, IDs by face, capable of measuring your pulse by sight
PrimeSense bought by Apple for $345M
Intel launched RealSense in 2014
Depth cameras on wearables: Meta Spaceglasses, Structure.IO
AI Audio interfaces
Cubic (Russian): “Let me become your personal assistant, home automation brain, consultant and a private coach.”
Amazon Echo
Eddy.io
6
http://cubicrobotics.ru/
A simple IoT scenario
Design the ultimate intelligent conference room coffee system
7
Centralized, decentralized – the pendulum swings
Centralized computing Distributed computing
Vacuum tube
Transistor
Intel 4004
Motorola 68000 Mainframe Client-server
Intel x86 Web server Thin client
Sun SPARC Mesh, peer-to-peer
ARM Cloud computing Mobile
Arduino, RPi IoT, Wearable
? Ambient, ubiquitous Implants, neural prosthetics…?
8
Why are AI chips emerging now?
Until now
Big data -> large datasets -> cloud computing
IoT devices -> very thin clients, basically sensors on a stick
Big centralized datasets to train compute-intensive AI
Volume, velocity… now variety
Diverse experiences are needed for AI to grow further
Geometric intelligence – learn in real time and react to novel experiences
AI shifting from centralized to distributed architecture
Sensor ubiquity at the edge
9
Past waves were winner-take-all. So who will be distributed AI’s winner, first loser, and worse?
Why AI has to move to the IoT Edge: the world is a messy place, full of novel (and not-so-novel) situations
April 29, 2016: “Summon … specifically mentions that the vehicle "may not detect certain obstacles" that are too low or too high for the car's sensors to see—perhaps why the car didn't stop before impacting the high-riding trailer.”
May 7, 2016: “The high ride height of the trailer combined with its positioning across the road and the extremely rare circumstances of the impact caused the Model S to pass under the trailer.”
10
Sources: http://cleantechnica.com/2016/07/02/tesla-model-s-autopilot-crash-gets-bit-scary-negligent/ http://www.roadandtrack.com/new-cars/car-technology/news/a29791/tesla-autopilot-fatal-crash-report/ http://www.roadandtrack.com/new-cars/car-technology/news/a29133/tesla-self-driving-crash-summon-autonomous/
AI is moving to the edge fast – via smartphones
Nexar, phone-based dashcam
Reads license plates and interprets images
Detects “hard brake” or accidents, automatically uploads video and data to the cloud – warns other users/drivers in the future 11
Many already see this opportunity: software/web giants open up AI tools
Google DeepDreams and TensorFlow
IBM opened up Watson
Facebook released its Torch tools
Microsoft open sourced its cross-server AI platform CNTK
Long-struggling Yahoo (yes, still alive) has entered the fray with CaffeOnSpark
Q: Why would these companies give away the future? A: Distributed learning, variety of experiences (i.e., not just dogs) will advance their AI efforts faster
12 Source: https://www.reddit.com/r/DeepDreaming/comments/3cemye/the_dog_is_watching/, https://www.reddit.com/r/deepdream/comments/3cb6vr/why_does_deep_dream_seem_to_have_an_enfatuation/
Q: Why does deep dream seem to have an infatuation with eyes and dogs? A: …these renders depend strongly on the statistics of the training data used for the ConvNet. In particular you're seeing a lot of dog faces because there is a large number of dog classes in the ImageNet dataset (several hundred classes out of 1000 are dogs)
Many already see this opportunity: Intel
Lost mobile to ARM – and does NOT want a repeat of that
Launching IoT-specific platforms like Basis, Curie and Galileo
Bought AI software companies Xtremeinsights (ML), Indisys (NLP), and Saffron (CC)
Spent a whopping $16.7 billion on AI chipmaker Altera
“Saffron offers a fresh look at big data analytics. We see an opportunity to apply cognitive computing not
only to high-powered servers crunching enterprise data, but also to new consumer devices that need to see, sense and interpret complex information in real
time. Big data can happen on small devices, as long as they’re smart enough and connected. Saffron’s
technology, deployed on small devices, can make intelligent local analytics possible in the Internet of
Things.”
Source: Intel blog, “Intel Acquires Saffron for Cognitive Computing” October 26, 2015
13
Source: Arrow development kit based on Altera MAX10 https://www.arrow.com/en/research-and-events/articles/max-10-fpgas-accelerate-the-design-of-cost-sensitive-iot-devices
Altera’s chips: field-programmable gate arrays (FPGAs) can be re-programmed after deployment.
“The Internet of things hook is one reason why Intel said it will continue to support Altera's ARM efforts. ARM will have a big chunk of the Internet of things market. Naturally, Intel will be integrating Altera's
wares with its Atom processor. But with Altera, Intel can play the Internet of things whether ARM or Atom
dominates. Source: ZDNet
Many already see this opportunity: Google
Google Tensor Processing Unit (TPU)
Application-specific integrated circuit (ASIC) for deep neural nets
Tied to TensorFlow
Buying Myriad VPU chips from Movidius
Smartphone AI – image recognition
Project Tango computer vision and 3D mapping project
“TPU is tailored to machine learning applications, allowing the chip to be more tolerant of reduced computational precision, which means it requires
fewer transistors per operation. Because of this, we can squeeze more operations per second into the
silicon, use more sophisticated and powerful machine learning models and apply these models more quickly,
so users get more intelligent results more rapidly.”
Source: Google blog, May 18, 2016
Fathom makes it easy to profile, tune and optimize your standard TensorFlow or Caffe neural network.
Fathom allows your network to run in embedded environments such as smart cameras, drones, virtual
reality headsets and robots. Fathom takes Deep Neural Networks to where they have never gone before, at high speeds and ultra-low power at the
network edge. Movidius is also introducing the Fathom Neural Compute Stick -- the first product of its kind -- a modular deep learning accelerator in the form
of a standard USB stick.
Source: Movidius
14 Source: http://www.pcworld.com/article/2464261/project-tango-chip-maker-movidius-touts-faster-second-gen-visual-processor.html
Many already see this opportunity: Nvidia
Tesla P100 GPU for deep learning and neural networks – but for data centers, not IoT
Jetson platform for autonomous machines (robots, drones, etc)
JetPack installs CUDA computing architecture on Jetson
“On top of CUDA is a library called cuDNN, which allows neural net developers to create their
frameworks to run as fast as possible. It lets you run DNNs 10-20x faster”
“Even compute-intensive video and image-processing applications, such as collision avoidance and
pedestrian detection”
Source: Nvidia
15
Startups to watch
Krtkl
Pi robotic chip
KnuEdge
Led by former NASA head Dan Goldin, raised $100 million and 100 employees
MIT Eyeriss
Deep-learning for speech recognition, face detection, object identification
Horizon Robotics
Led by Kai Yu, former head of Baidu’s Institute of Deep Learning
Targeting vehicle safety and self-driving vehicles.
Nervana
Working on neuromorphic chips, but making software-based applications in the meantime
TeraDeep
Runs on conventional hardware, even old stuff
16
Many already see this opportunity: Everyone in the sensor value chain
17
$85 million
$3.2 billion
$92 million
$36 million
$1.4 billion
$20 million
$30 million
$450 million
Co
nsu
mer
M
edic
al
Ag
In
du
stri
al
W
eara
ble
s A
uto
mo
tive
Undisclosed
Undisclosed
Undisclosed
Over 780 transactions, $4.3 billion was invested over a 10-year period
Over 780 transactions, $4.3 billion was invested in sensor developers since 2006. The investments grew from about $180 million in 2006 to over $625 million in 2015, a CAGR of over 13%. The drop in investment in 2009 coincided with the global financial crisis. The drop in investment dollars in 2012 is more an artifact of the data available (several transaction prices were undisclosed).
In the first four months of 2016, $236 million has already been invested, which represents about 38% of the total investments in 2015. As such, 2016 is on track to seeing an upward trend for sensor developers to attract investment.
18
0
20
40
60
80
100
120
140
0
100
200
300
400
500
600
700
800
20
06
20
07
20
08
20
09
20
10
20
11
20
12
20
13
20
14
20
15
20
16
Nu
mb
er
of
tra
nsa
ctio
ns
Inve
stm
en
t ($
mil
lio
n)
Total VC investments 2006-2016
Transaction value Number of transactions
*
Building and heavy industry invested $1.8 billion; auto second to last at $288 million
Sensor innovations targeted at the building and heavy industry sector attracted the lion’s share of the investment, at $1.8 billion since 2006. This included investments in oil-and-gas-related sensors, sensors for water quality monitoring, or building-efficiency related sensor innovations.
Sensor innovations for the consumer sector attracted about $1.1 billion since 2006. This included developers of fingerprint sensors for mobile devices, image sensors, wireless wearable sensors as well as developers of processors for mobile and wearable applications.
Sensors for medical/human health applications attracted about $780 million overall, while sensors for automotive applications attracted about $288 million. Sensors for food and agriculture have seen an influx of about $250 million since 2006.
19
$-
$100.00
$200.00
$300.00
$400.00
$500.00
$600.00
$700.00
$800.00
2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Inve
stm
en
t ($
mil
lio
n)
Building & heavy industry Food & agriculture Medical Consumer Automotive
*
“We do not plan to become the Foxconn of Apple” Dieter Zetsche, CEO Daimler Sept 17, 2015
“In 2007 I pledged that – by 2010 – Nissan would mass market a zero-emission
vehicle. Today, the Nissan LEAF is the best-selling electric vehicle in history. Now I am
committing to be ready to introduce a new ground-breaking technology,
Autonomous Drive, by 2020, and we are on track to realize it.”
“We have seen what Google did to phone manufacturers, and we don’t want that to
happen to us.”
-Nissan CEO Carlos Ghosn
The auto industry utterly failed in telematics; will they repeat?
Carmakers must make networked vehicles now, to prevent the rise of third-tier OEMs
Consortia are collapsing into proprietary, competitive programs
http://www.nytimes.com/2015/09/18/automobiles/apples-auto-inroads-create-a-buzz-at-frankfurt-motor-show.html?ref=technology&_r=2 20
Carmakers have seen the threat and are responding
21
The improving ecosystem (not the tech) pushed OEMs to adopt Bluetooth/phone connectivity
Bluetooth was invented by Ericsson in 1994, and saw early adoption in mobile phones beginning in 2000. It has become the “connective glue” between many diverse technologies, including cars
The technology started as a way for drivers to call people on mobile phones without using their hands (hands-free).
In the mid to late 2000s, Bluetooth syncing features evolved past hands-free and into integration with apps and telematics services.
22
Bluetooth hands-free integration:
Bluetooth app integration:
New smartphone-enabled business models provide opportunities for drivers that OEMs missed
In recent years, there has been a rise in companies offering new business models to compete with traditional auto sales. Smartphone connectivity – not OEM telematics - is key to most of them
This change in how people pay for and use vehicles points to the trend of consumers looking for product offerings that cater to their needs and personal preferences, even in a car that will be shared by multiple users.
Having missed the opportunity to control the technology, automakers are reacting: changing from “auto-manufacturers” to “mobility-providers”
23
Cars, connectivity and new business models, Chapter 2: OBD dongles
Car-sharing and ridesharing companies provide drivers with a cars-as-a-service alternative to paying for a vehicle and access to an entire fleet of cars, and even presents an alternative for accessing these cars by replacing a physical key with an app on a smartphone.
24
Traditional Car Sales New Business Models
Developers like Automatic (see the February 24, 2016 LRASJ) and Mojio (see the December 3, 2014 LRASJ), which develop OBD-II port dongles, leverage vehicle diagnostic data and connectivity tools to offer additional services to drivers, such as predictive maintenance and alternatives for insurance.
Cars, connectivity and new business models, Chapter 3: Intelligent transportation systems (ITS)
ITS BG (Before Google)
V2V, V2I in nationally-defined systems
ITS AG (After Google)
autonomy based on onboard, real time image analytics
25
Cars connected to other cars, infrastructure, and driver via
Bluetooth, WiFi, 5G…
Cars connected to other cars, infrastructure, and driver via
vision (images + AI)
Integrating autonomy – where can image data be the connective glue?
26 Technology Readiness
Te
chn
olo
gy
Att
ract
ive
ne
ss
Scale Deployment Concept Stage
Mo
st a
ttra
ctiv
e L
east
att
ract
ive
Lux Research Inc. 100 Franklin Street, 8th Floor Boston, MA 02110 USA Phone: +1 617 502 5300 www.luxresearchinc.com
Thank you
Mark Bünger, VP of Research