microsoft streaminsight
DESCRIPTION
Microsoft StreamInsight. Allan Mitchell Konesans Limited. Who am I?. Allan Mitchell Joint author on “First to Market” SSIS Book by Wrox Websites www.SQLIS.com www.SQLDTS.com www.konesans.com StreamInsight and SQL Azure Advisory Boards Microsoft SQL Server MVP Twitter : allanSQLIS - PowerPoint PPT PresentationTRANSCRIPT
Microsoft StreamInsight
Allan MitchellKonesans Limited
Who am I?• Allan Mitchell• Joint author on “First to Market” SSIS Book by Wrox• Websites
– www.SQLIS.com – www.SQLDTS.com– www.konesans.com
• StreamInsight and SQL Azure Advisory Boards• Microsoft SQL Server MVP• Twitter: allanSQLIS• eMail: [email protected]
Agenda
• What is Stream Processing• Understanding Streaming Data
– Differences to RDBMS• Scenarios• StreamInsight Architecture• Concepts• Typical Patterns
What is Event Stream Processing
Query
• Database loads data from disk• You query database• Data is stale
What is Stream Processing
• Continuous processing of event streams• from multiple sources• based on a declarative query language• with near-zero latency
StreamInsight
Query
Understanding Streaming Data
Using an RDBMS:
Q. Given a time interval tell me the amount of Electricity units used per household per time interval from now until I say stop.
Not EasyRequires requerying/polling
Need to store the end of the last interval
Understanding Streaming Data
This is the streaming data paradigm in a nutshell – ask questions about data in flight.
Answering with StreamInsight
• Create a window over the event stream• Partition/Group By Household• SUM() readings within window
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
What is CEP?
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
What is CEP?
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or less
What is CEP?
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or lessData Query Rate
Hundreds of events/sec
> Tens of thousands of events/sec
What is CEP?
Relational Database Applications
Financial trading Applications
Scenarios for Event-Driven Applications
12
Aggregate Data Rate (Events/sec.)
Latency
0 10 100 1000 10000 100000 ~1million
Months
Days
hours
Minutes
Seconds
100 ms
< 1ms
Operational Analytics Applications, e.g., Logistics,
etc.
Manufacturing Applications
Monitoring Applications
StreamInsight Target Scenarios
Data Warehousing Applications Web Analytics Applications
Who might need CEP
• Fraud Detection• Real-Time Trade Risking• Algorithmic Trading/Betting• Meter throughputs
– Oil, Gas, Water, Electricity– Use to drive alarms, alerts etc
• Telemetry Data
Windowing
• Hopping– Tumbling
• Snapshot• Count
Time Windows
Time
Hopping Window
Time Windows
Time
Tumbling Window
Snapshot Windows
Snapshot Windows
StreamInsight Key Features• Runtime environment for continuous and event-driven
processing• In-memory processing, no persistence• Declarative query language: LINQ• .NET APIs for Input/Output Adapters• Flexible deployment models• Extensibility: Aggregates, Operators• Rich diagnostic & debugging interface
Standing Queries
Query Logic
Event sources Event targets
`
Devices, Sensors
Web servers
Event stores & Databases
Stock ticker, news feeds
Event stores & Databases
Pagers &Monitoring devices
KPI Dashboards, SharePoint UI
Trading stations
StreamInsightApplication Development
InputAdapter
s
OutputAdapter
sStreamInsight
Engine
Query Logic
Query Logic
StreamInsight Platform
12 3
4
StreamInsight Application at Runtime
Microsoft StreamInsight
21
CEP Engine
Output Adapters
Input Adapters
Event
Standing Queries
Event sources Event targets
`
Event
Event
Event
Event
Event
Event
C_ID C_NAME C_ZIP
Event
Event
Static reference data
CEP Application Development
Development experience with .NET, C#, LINQ and Visual Studio 2008/10
CEP platform from Microsoft to build event-driven applications
Event-driven applications are fundamentally different from
traditional database applications: queries are
continuous, consume and produce streams, and compute
results incrementally
Flexible adapter SDK with high performance to
connect to different event sources and sinks
The CEP platform does the heavy lifting for you to deal
with temporal characteristics of event stream data
Central Concepts• Events
– User Payload + temporal properties– Type based on .NET structure or class– Point / Interval / Edge
• Streams– Sequence of events with specific type– Flow into standing queries
• Queries– Composed of query operators– Apply desired semantics on events– In-memory, event-triggered, incremental computations
• Adapters– Produce / Consume event streams (Connection Manager)
Typical Query & Analytics Patterns
• Introduce additional event fields• Filter out events• Grouping by one or more event properties• Aggregation for each event group over a set (a window)• Rank events within windows• Correlate and synchronize event streams• Check for absence of events• Create result only when input changes• Remove duplicates• Reduce event rate, resample stream• Enrich events with reference data• Collection of assets may change over time
LINQ Query Examples
LINQ Example – GROUP&APPLY, WINDOW:from e3 in MyStream3group e3 by e3.i into SubStreamfrom win in SubStream.HoppingWindow( FiveMinutes,ThreeSeconds)select new { i = SubStream.Key, a = win.Avg(e => e.f) };
LINQ Example – JOIN, PROJECT, FILTER:from e1 in MyStream1join e2 in MyStream2 on e1.ID equals e2.IDwhere e1.f2 == “foo”select new { e1.f1, e2.f4 };
Join
FilterProject
Grouping
WindowProject &Aggregate
Extensibility
• Need for domain-specific extensions– Integrate with functionality available in existing libraries
• User-defined operators, functions, aggregates• Code written in .NET, deployed as .NET assembly• Window-based semantics for UDOs, UDAs
– Receive a set of events– Produce a value / set of events
Demo
• Adapter model• Windows• Grouping• Altering event lifetime• Extending
For More Information
• Download:http://go.microsoft.com/fwlink/?LinkId=160598
• StreamInsight MSDN documentation:http://msdn.microsoft.com/en-us/library/ee362541(SQL.105).aspx
• Blogs:http://blogs.msdn.com/streaminsight/http://www.SQLIS.com/
• Forum:http://social.msdn.microsoft.com/Forums/en-US/streaminsight
• StreamInsight MSDN portal:http://msdn.microsoft.com/en-us/ee476990.aspx
• Samples:http://streaminsight.codeplex.com
• Twitter:http://twitter.com/streaminsight