moving targets: harnessing real-time value from data in motion
TRANSCRIPT
Grab some
coffee and
enjoy the
pre-show
banter before
the top of the
hour!
The Briefing Room
Moving Targets: Harnessing Real-Time Value from Data in Motion
Twitter Tag: #briefr The Briefing Room
Reveal the essential characteristics of enterprise software, good and bad
Provide a forum for detailed analysis of today’s innovative technologies
Give vendors a chance to explain their product to savvy analysts
Allow audience members to pose serious questions... and get answers!
Mission
Twitter Tag: #briefr The Briefing Room
Topics
February: DATA IN MOTION
March: BI/ANALYTICS
April: BIG DATA
Twitter Tag: #briefr The Briefing Room
Parmenides and the Truth of Now
There is no tomorrow
There is no yesterday
There is only today
There is only now
Twitter Tag: #briefr The Briefing Room
Analyst: David Loshin
David Loshin, president of Knowledge Integrity, Inc, is a thought leader and expert consultant in the areas of data quality, master data management, and business intelligence. David is the author of numerous books and papers on data management, including the “Practitioner’s Guide to Data Quality Improvement.” David is a frequent speaker at conferences and in web seminars. His best-selling book, “Master Data Management,” has been endorsed by data management industry leaders. David can be reached at [email protected], or at (301) 754-6350.
Twitter Tag: #briefr The Briefing Room
Datawatch
Datawatch began as a BI tool and has developed into a visual analytics platform
The platform provides visual data analytics and discovery on any type of data, including streaming data
The suite of products are Datawatch Desktop, Datawatch Server, Datawatch Report Mining Server and Datawatch Modeler
Twitter Tag: #briefr The Briefing Room
Guest: Dan Potter
Dan Potter is the Vice President of Product Marketing at Datawatch Corporation. In this role, Dan leads the product marketing and go-to-market strategy for Datawatch. Prior to Datawatch, Dan held senior roles at IBM, Oracle, Progress Software and Attunity where he was responsible for identifying and launching solutions across a variety of emerging markets, including cloud computing, visual data discovery, real-time data streaming, federated data and e-commerce.
VISUAL DATA DISCOVERY & STREAMING DATA New Technologies for Real-Time Analytics
Dan Potter Vice President, Product Marketing
NASDAQ: DWCH Pioneer in real-time visual data discovery and self-service data preparation
Global operations and support § US, UK, Germany, France, Australia, Singapore, Philippines
Extensive global customer base § 93 of the Fortune 100 § 12 of the 15 largest financial institutions
Embedded and resold by leading vendors
About Datawatch
DISCOVER
GOVERN
ACQUIRE
PREPARE
AUTOMATE
Visual Analytics Platform For Any Data at Any Speed
Where Do Real-Time Streams Come From?
• Internet of Things • Machine data / log files • Web clickstreams • Enterprise applications • Human generated • Commercial data
Streaming Visualization Examples
Capital Markets § Transac'on Cost Analysis § Analyze market data at
ultra-‐low latencies § Momentum Calculator
Fraud preven2on § Detec'ng mul'-‐party fraud § Real 'me fraud preven'on
e-‐Science § Space weather predic'on
§ Detec'on of transient events § Synchrotron atomic research
§ Genomic Research
Transporta2on § Intelligent traffic
management § Automo've Telema'cs
Energy & U2li2es § Transac've control
§ Phasor Monitoring Unit § Down hole sensor monitoring
Natural Systems § Wildfire management § Water management
Other § Manufacturing
§ ERP for Commodi'es
§ Real-‐'me mul'modal surveillance § Situa'onal awareness
§ Cyber security detec'on § Emergency Evacua'on
Law Enforcement, Defense & Cyber Security
Health & Life Sciences
§ ICU monitoring § Epidemic early warning § Remote healthcare
monitoring
Telephony § CDR processing § Social analysis
§ Churn predic'on § Geomapping
Visual Data Discovery
• Easy for users to author, customize and share
• Interactive exploration & visually filter results
• Quickly identify anomalies and outliers with large or in-motion datasets
• Rich palette of visualizations for static and time series data
Visualize Any Data at Any Speed
Stream Rela2onal NoSQL OLAP Warehouse Hadoop Content
Connect, Federate, Visualize
Data Architectures Evolving
Database Distributed or Hybrid Database
In-‐Memory Database
Streaming Analy'cs
Faster Speed, Faster Insights
Data at Rest
Limitations of Traditional BI
Database Distributed or Hybrid Database
In-‐Memory Database
Streaming Analy'cs
Data at Rest
Streaming Data Visualization
Database Distributed or Hybrid Database
In-‐Memory Database
Streaming Analy'cs
Datawatch Streaming Data Visualization
• Connect directly to data in motion • CEP (IBM Streams, Informatica Rulepoint, Tibco Streambase) • Hosted IoT platforms (Amazon Kinesis, PTC ThingWorx) • Message Bus (Informatica UltraMessaging, WebSphere MQ) • Operational Intelligence Systems (OSIsoft Pi)
• Purpose built data model optimized for both caching and persistence
• High density visuals with rendering in milliseconds
Monitor
Analyze
Take Ac2on
Time Series Data • Traditional BI only looks at buckets of
time • Day, week, month, year
• Streaming data is a continuous and has different requirements
• Second, millisecond, nanosecond • Time windows • Time slices • Playback
• Complete situational awareness • Now (streaming) • Intra-day • Historic
Predictive & Advanced Analytics
• Connect to R (Rserv) and Python (Pyro) servers
• Transform using R and Python
• Many use cases in IoT (e.g. predictive maintenance, smart logistics, clinical pattern detection etc.)
Modeled and transformed for analysis
Complex File Formats
• Sensor and machine data often in multi-structured format • Need to transform, enrich and prepare data
• Almost no metadata • For example, wave form visualization from JSON arrays
stored in MongoDB and streaming
23
Log Files
HTML, XML JSON
PDFs
Real-Time Geospatial & Location
• Real-time (stream) plotting • Street-level geo maps or
custom SVG files • Time-series playback
Healthcare Retail
Logis'cs
U'li'es
Customer Challenge
Dozens of risk management systems generating data silos of operational information
Server based solution to visualize integrated risk information in real-time to identify trends and anomalies
Analyze patterns in physiological data that may detect and eventually to predict deadly clinical events
Visualize large volumes of streaming, unstructured data from multiple devices in real-time
Improve yield production and enhance machine reliability in contact lens manufacturing process
Flexible visualization solution highlighting production line yield, leading to a 2% yield increase and 750,000 additional units produced
Real-World, Real-Time Examples
Process and visualize billions of streaming trades per day for leading surveillance and compliance platform
Fully embedded visual data discovery solution that delivers a single consolidated real-time view of trading across venues
Twitter Tag: #briefr The Briefing Room
Perceptions & Questions
Analyst: David Loshin
Brie%ing Room 02-‐17-‐2015: Considerations for Streaming Analytics
2015-‐02-‐17 David Loshin
Knowledge Integrity, Inc. loshin@knowledge-‐integrity.com
© 2015 Knowledge Integrity, Inc loshin@knowledge-‐integrity.com (301) 754-‐6350 28
Technology Convergence & Stream Analysis
• Discovery & Streaming Analy'cs employs a number of key evolving technologies beyond the expected “repor'ng & analy'cs”: – Data virtualiza'on and federa'on – Text parsing and text analy'cs – Seman'c models – Real-‐'me data inges'on – Event stream processing – Embedded rules for monitoring, no'fica'on, and alerts – In-‐memory processing – Visualiza'on
• Con'nued improvements in these technologies will automa'cally improve the quality and speed of real-‐'me stream analy'cs
© 2015 Knowledge Integrity, Inc loshin@knowledge-‐integrity.com (301) 754-‐6350
29
Future Direction of Connected Devices?
• More “things” will be networked – Who’d a thunk that thermostats
would be in the first wave of smart devices?
• Networked “things” will be gegng “smarter” – More & beher resources at the
device • Increased open-‐source
standardiza'on – Including the hardware!
• Increased ease of programmability expands the community of developers – A 12-‐year old can program this
© 2015 Knowledge Integrity, Inc loshin@knowledge-‐integrity.com (301) 754-‐6350
30
Considerations
• The volume and variety of human-‐generated content will con'nue to explode – This will require increased analy7c intelligence for parsing and filtering
within the network
• Par'al analy'c computa'ons can be pushed out to the devices – Move the applica7on to the data, not the data to the applica7on
• Alerts and no'fica'ons base on the results of intermediate analyses can provide advantage in mul'ple ways – The same data streams can feed a wide variety of consumer communi7es
• Streaming analy'cs will become relevant at the personal as well as the business level – Enable personalized algorithmic stream blending, analysis, and
monitoring/no7fica7on at the mobile device
© 2015 Knowledge Integrity, Inc loshin@knowledge-‐integrity.com (301) 754-‐6350
31
Questions to Explore
• What has predicated the growth in demand for analyzing streaming data in recent years?
• What are the types of streaming data that are most frequently subjected to analysis?
• What are the features of your product that have been most valuable to your customer community, and why?
• How does your product help business users dis'nguish relevant streaming content from the “noise”?
• Can you share some insight into how your tool uses in-‐memory processing and manages data in memory?
• What fundamental differences do you see between the ability to enable analysis of human-‐generated content vs. machine-‐generated streaming content?
• Can you share thoughts about external constraints that prevent the best opportuni'es for using streaming analy'cs and discovery?
• What do you see as the next hurdles in enabling business consumers in adop'ng discovery analy'cs for streaming data?
• Who are the compe'tors and what do you see as the advantages your tool provides over your compe'tors?
© 2015 Knowledge Integrity, Inc loshin@knowledge-‐integrity.com (301) 754-‐6350
32
Following up…
• www.knowledge-‐integrity.com • www.dataqualitybook.com • www.decisionworx.com • If you have ques'ons, comments,
or sugges'ons, please contact me David Loshin 301-‐754-‐6350 loshin@knowledge-‐integrity.com
© 2015 Knowledge Integrity, Inc loshin@knowledge-‐integrity.com (301) 754-‐6350
33
Twitter Tag: #briefr The Briefing Room
Twitter Tag: #briefr The Briefing Room
Upcoming Topics
www.insideanalysis.com
February: DATA IN MOTION
March: BI/ANALYTICS
April: BIG DATA
Twitter Tag: #briefr The Briefing Room
THANK YOU for your
ATTENTION!
Some images provided courtesy of Wikimedia Commons and Wikipedia