mark davis - ieee computer societymedia.computer.org/pdfs/davis.pdf · mark davis distinguished...
TRANSCRIPT
![Page 1: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/1.jpg)
Mark Davis Distinguished Engineer, Dell Software Group
IEEE Cloud Computing Initiative
IEEE Intercloud Interoperability Testbed
![Page 2: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/2.jpg)
Industries Vertical domains Concepts and Algorithms Technologies
![Page 3: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/3.jpg)
200EB = 1018 B
1ZB = 1021 B
10EB
100TB
2000 1985 1900 1750
Industrial
Revolution
#1
Industrial
Revolution
#2
Industrial
Revolution
#3
Industrial
Revolution
#4
R. J. Gordon: Is US economic growth over? Faltering innovation confronts the six headwinds. CEPR Policy Insight No 63
Krugman, P. Is Growth Over? New York Times, 12 December 2012.
![Page 4: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/4.jpg)
I visualize a time when we will be to
robots what dogs are to humans, and
I'm rooting for the machines.
Claude Shannon, IEEE Medal of
Honor 1966
![Page 5: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/5.jpg)
![Page 6: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/6.jpg)
![Page 7: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/7.jpg)
Faster reporting
Interactive visualization
$$$
Data warehousing
Advanced analytics
Just-in-time decisionmaking
The Perl Scripting option
![Page 8: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/8.jpg)
4TB of data on disk
4 channels of I/O at 100Mb/s
2.7 hours
![Page 9: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/9.jpg)
Server farms are too expensive
Oracle is too expensive
We just need key-value stores
VCs say no CAPEX
Be lean, young entrepreneurs, be lean
![Page 10: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/10.jpg)
![Page 11: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/11.jpg)
![Page 12: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/12.jpg)
Analyze log files from web
servers
Group users based on behavior patterns
Build machine learning
models of behavior
Recommend news and information
Social Media
![Page 13: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/13.jpg)
Collect millions of data
points from sensors
Analyze failure modes
Proactively improve products
Optimize systems
Industrial Control
![Page 14: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/14.jpg)
Defense Intelligence
![Page 15: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/15.jpg)
Anomaly Detection
![Page 16: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/16.jpg)
Scientific Analysis and Visualization
![Page 17: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/17.jpg)
Collective Intelligence
![Page 18: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/18.jpg)
Social Network Analysis
![Page 19: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/19.jpg)
Collective Intelligence
Anomaly Detection
Distribution Network Optimization
Scientific Analysis and Visualization
Spam Filtering
Defense Intelligence
Product Design
Internet of Things
Social Network Analysis
Search
![Page 20: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/20.jpg)
Linguistic
Understand language
Use linguistic concepts to create computational systems
Understand
Statistical
Data driven
Language is structured information
![Page 21: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/21.jpg)
Hybrid
Understand language some
Use statistics and machine learning to help with recall
![Page 22: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/22.jpg)
Parts-of-Speech Tagging
Tokenization
Lemmatization
Finite State Transducer Finite State Transducer
Finite State Transducer
Machine-Learning
![Page 23: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/23.jpg)
![Page 24: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/24.jpg)
Random Indexing:
Assign random, sparse vectors to words
(or entities)
Add together the vectors for a context
Related contexts cluster in high-
dimensional space
Assign new metadata from discovered
clusters to documents
![Page 25: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/25.jpg)
Why? Model for human/animal sparse distributed memory Success in the TOEFL test (64.5-67%) Distributional hypothesis: words with similar co-occurrence patterns
have similar meanings Johnson-Lindenstrauss Lemma: projecting a matrix through a random
matrix preserves the relative distances between points if R is high dimensionality
Fixed context vectors (say, 4000 bits) reduce complexity Versatile: words-words contexts, words-document contexts, entity-
entity contexts, x-y contexts
![Page 26: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/26.jpg)
MapReduce RI requires reduce phase that merges random spaces
Or…Precompute sparse word vectors Or…Serve the map phase from a common
term signature service Or…Same hash function across instances and
compute a sparse vector (1% occupancy)
![Page 27: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/27.jpg)
Clustering of terms, entities, or documents Autosuggestion Categorization Abstractly: generalized similarity engine
![Page 28: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/28.jpg)
![Page 29: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/29.jpg)
Query Language
Metadata Extraction
Indexing
Facet Browsing Facet Charting
Resource Integration
Autosuggest Spellcheck
![Page 30: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/30.jpg)
![Page 31: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/31.jpg)
Big Data search and analytics has many challenges: Volume of data Variety of data Velocity of data Extracting structure from unstructured information:
▪ Machine Learning ▪ Human Intelligence ▪ Knowledge Engineering
Enabling the 21st Century
![Page 32: Mark Davis - IEEE Computer Societymedia.computer.org/pdfs/davis.pdf · Mark Davis Distinguished Engineer, Dell Software Group IEEE Cloud Computing Initiative IEEE Intercloud Interoperability](https://reader035.vdocuments.us/reader035/viewer/2022062606/5feb0cc43ca7bd622d050b64/html5/thumbnails/32.jpg)