a picture’s worth a thousand hashtags: how image recognition will power the future of analytics
TRANSCRIPT
![Page 1: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/1.jpg)
A picture’s worth a thousand hashtags:
How image recognition will power the future of analyticsDavid BerkowitzChief Strategy [email protected]@sysomos / @dberkowitz
![Page 2: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/2.jpg)
About this presentation
This talk was presented to Marketing Week Live in London in March 2017. A Texan version of this was delivered to W2O Group’s Pre-Commerce Summit during SXSW that month. If you prefer Frito pie to bangers and mash, I will gladly send you the W2O edition.
Sources for information shown in slides are presented as links in the bottom-left corner. Further details about sources, where applicable, are also included in the notes field when downloading this presentation.
To share your feedback or discuss this further, please contact me at [email protected].
Thank you.
David
![Page 3: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/3.jpg)
And now for something completely standard: an agenda
• A brief history of nearly everything visual search
• Why visual search matters
• How Google, Pinterest, and others are deploying it
• How marketers can use it
• Numerous gratuitous references to all things British
![Page 4: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/4.jpg)
?
![Page 5: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/5.jpg)
Not British, but just a Chunnel ride away
![Page 6: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/6.jpg)
How would you #HASHTAG
the Mona Lisa?
![Page 7: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/7.jpg)
#YOLO
![Page 8: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/8.jpg)
Mona Lisa #Hashtags
• #MonaLisa
• #art
• #painting
• #woman
• #lady
• #smile
• #smug
• #Italian
• #DaVinci
• #epic
• #outdoors
• #masterpiece
• #sky
• #France
• #LaGioconda
• #badhairday
• #beautiful
• #Louvre
• #Renaissance
• #portrait
This is a futile exercise. One can’t simply capture the Mona Lisa in hashtags. It points to the need for better ways to
identify and analyze visual content. Text and hashtags alone don’t cut it.
![Page 9: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/9.jpg)
Textual Media
Visual Media
We are here.
![Page 10: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/10.jpg)
Drop in a bucket visual
The Visual Data GapRecall how pictures are worth a thousand words? There is
so much more data in images (‘the sun’) than there is in text (represented metaphorically by the planets).
![Page 11: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/11.jpg)
A Short History of Nearly Everything Visual
![Page 12: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/12.jpg)
Dual British/ American citizen
![Page 13: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/13.jpg)
A challenge of Shakespearean proportions:
“His reasons are as two grains of wheat hid in two bushels of chaff: you shall seek all day ere you find them, and when you have them, they are not worth the search.”-Bassanio, The Merchant of Venice
![Page 14: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/14.jpg)
Number of object categories out there
15,000There are 15,000
object categories
Source: IEEE
![Page 15: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/15.jpg)
Source: Computer Vision by Richard Szeliski
For fun, I included a few examples of early attempts at machine-
powered object recognition.
![Page 16: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/16.jpg)
Source: Computer Vision by Richard Szeliski
![Page 17: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/17.jpg)
An inflection point waiting to happen
Jason Goldberg, Razorfish: “I’m strongly bullish on visual search. It solves a real problem consumers have… In the not-too-distant future, it’ll become a heavily used mainstream feature. I think the inflection point is at least a year away, but not two years." We’re approaching the
inflection point, but it has taken longer than expected.
This report is from November 2014.
![Page 18: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/18.jpg)
Scanning products = Cool
Facial scanning = Creepy
![Page 19: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/19.jpg)
This is simply an enlarged, cropped
version of the highlights from the
previous chart.
![Page 20: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/20.jpg)
You can’t always get what you want (with text search)
• 74% of consumers say text-based keyword searches are inefficient for helping them find the right products online
• 67% of consumers say quality of product images is very important in selecting and purchasing products
• 90% of information transmitted to the brain is visual
• Visual information is processed 60,000x faster than text
Source: Slyce
![Page 21: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/21.jpg)
The dress that inspired Google Image Search in 2001
![Page 22: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/22.jpg)
The dress that inspired Google Image Search in 2001
“…People wanted more than just text. This first became apparent after the 2000 Grammy Awards, where Jennifer Lopez wore a green dress that, well, caught the world’s attention. At the time, it was the most popular search query we had ever seen. But we had no surefire way of getting users exactly what they wanted: J Lo wearing that dress. Google Image Search was born.”-Eric Schmidt, Executive Chairman, Google
Source: Project Syndicate
![Page 23: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/23.jpg)
Solving the Clarissa problem My wife gave me this reference. As a kid, she always wanted to identify and shop for whatever Clarissa wore.
![Page 24: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/24.jpg)
Applications for image recognition
Source: Facebook
![Page 25: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/25.jpg)
Source: The Verge
![Page 26: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/26.jpg)
An introductory framework for visual search
![Page 27: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/27.jpg)
Layers of image recognition
A Deep Learning algorithm is presented with the images made up of simple pixels.
The algorithm discovers simple regularities that are present across many/all images, like curves & lines.
The algorithm discovers how these regularities are related to form higher-level concepts
The system gains a high-level understanding of the original image… all automatically
Source: GazeMetrix
![Page 28: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/28.jpg)
A framework for visual search
Scene
Identification Intelligence
ObjectIdentification Intelligence
LogoIdentification Intelligence
ImageIdentification Intelligence
CategoryIdentification Intelligence
This notes some of the most important processes within visual search. Also note that identification and intelligence
are two separate approaches. Examples follow.
![Page 29: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/29.jpg)
What follows is an example using a real photo from Agnes, a
Chinese tourist to the UK.
![Page 30: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/30.jpg)
Here’s her photo. In each subsequent slide, you can see how the framework
plays out and a sample finding that can be derived. Note the intelligence
examples that follow are for illustrative purposes only; feel free to cite the framework, but not the data itself.
![Page 31: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/31.jpg)
Category Identification:This is food and drink
Category Intelligence:7.2% of images posted at museums include food or
drinks
![Page 32: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/32.jpg)
Logo Identification:There is a Fanta logo, and
the text in the top-right says Starbucks
Logo Intelligence:Fanta logos are rarely paired with Starbucks; Fanta logos
are most often seen with Coca-Cola and Adidas
![Page 33: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/33.jpg)
Object Identification:This looks like fish and chips with a can of Fanta Lemon
Object Intelligence:Fanta Lemon is the fourth most popular soda when paired with fish & chips
![Page 34: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/34.jpg)
Image Identification:This is the same photo that
appears on Agnes_Cin’s Flickr and public Facebook
pages
Image Intelligence:This image hasn’t been
shared in any media outlets and hasn’t been shared
publicly
![Page 35: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/35.jpg)
Scene Identification:This photo seems to be taken
outdoors during the day
Scene Intelligence:94% of photos at the British Museum are shot indoors,
compared to 87% of museum photos worldwide
![Page 36: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/36.jpg)
Spotlight: Google
![Page 37: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/37.jpg)
This section is drawn (no pun intended) from quickdraw.withgoogle.com.
![Page 38: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/38.jpg)
![Page 39: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/39.jpg)
![Page 40: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/40.jpg)
![Page 41: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/41.jpg)
![Page 42: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/42.jpg)
Spotlight: Pinterest
![Page 43: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/43.jpg)
Pinterest: one of the world’s biggest search engines
• 150 million monthly users• 75 billion pins• 2 billion searches/month• 97% of searches are unbranded
Source: Pinterest
![Page 44: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/44.jpg)
Pinterest: search pins from real-world images
Source: Pinterest
Examples from Pinterest’s new Visual Discovery follow. In the downloadable
version of this talk, the next few visuals play as GIFs, and you can read more at
the source below.
![Page 45: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/45.jpg)
Pinterest Visual Discovery
Source: Pinterest
![Page 46: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/46.jpg)
Browse images and buy from them
Source: Pinterest
![Page 47: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/47.jpg)
![Page 48: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/48.jpg)
![Page 49: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/49.jpg)
![Page 50: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/50.jpg)
![Page 51: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/51.jpg)
I really liked the circled image here: Pinterest draws a connection between Big Ben and the Blue Mosque. Such
errors are often more revealing.
![Page 52: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/52.jpg)
Spotlight: Emerging Technologies
![Page 53: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/53.jpg)
Toys R Us offers Slyce image detection for its catalog
Source: Slyce
![Page 54: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/54.jpg)
Visual search to complement textual search
eMarketer: Do you think [visual search] will replace some types of searches, or do you think it will augment existing searches?
Gierhart: It will probably augment. It’s adding a new utility to what was there before... There will still be contexts for both.
Source: eMarketer (see a related video on YouTube)
![Page 55: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/55.jpg)
Blippar: scan images for surprises about Planet Earth
Source: Blippar
![Page 56: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/56.jpg)
Source: Houzz
Houzz has a Visual Match offering akin to
Pinterest.
![Page 57: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/57.jpg)
TheTake uses AI to identify products, locations in video
Source: TheTake
![Page 58: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/58.jpg)
What’s Possible with Image Analytics
The images that follow are sample reports drawn from Sysomos. The data is again for illustrative purposes. Reach out if you want to dive deeper into any of
this.
![Page 59: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/59.jpg)
Visual analysis: understanding visual characteristics
Logo recognition
Object recognition
Scene recognition
Food recognition
Color detection
OCR: Search text within images
![Page 60: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/60.jpg)
Visual analysis: understanding visual characteristics
Logo recognition
Object recognition
Scene recognition
Food recognition
Color detection
OCR: Search text within images
![Page 61: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/61.jpg)
Visual analysis: understanding visual characteristics
Logo recognition
Object recognition
Scene recognition
Food recognition
Color detection
OCR: Search text within images
![Page 62: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/62.jpg)
Audience analytics show growth and spikes
![Page 63: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/63.jpg)
Brand affinities can highlight cross-promotion opportunities
![Page 64: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/64.jpg)
Identify most popular objects, scenes
![Page 65: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/65.jpg)
B2B applications for visual search
Creative optimization
Influencer marketing
Rights manageme
nt
Crisis manageme
nt
Partnership ideation
Competitive intelligence
Customer service
![Page 66: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/66.jpg)
Any questions?
![Page 67: A Picture’s Worth a Thousand Hashtags: How image recognition will power the future of analytics](https://reader035.vdocuments.us/reader035/viewer/2022070600/58ce75421a28abdc578b6579/html5/thumbnails/67.jpg)
So long, and thanks for all the fish.Let’s take tea!
David BerkowitzChief Strategy [email protected]@sysomos / @dberkowitz