real-time event processing using linq for logs & traces
TRANSCRIPT
Real-time Event Processing using LINQ for Traces (TX)Gert Drapers (#DataDude)
Principle Software Design Engineer
Agenda
•Rx (Reactive Extensions)
•Tx (LINQ to Logs and Traces)
•Case Study
Subtitel
TX (LINQ to Logs and Traces)
•http://tx.codeplex.com/
•TX allows you use LINQ directly on raw event sources:• Ad-hoc query on past history in
trace and log files • Standing queries on real-time feeds,
such as Event Tracing for Windows (ETW) sessions
Reactive Essential Interfaces
namespace System {
// Producers (.NET events, WinRT Events, Sensors, APM method etc.)
public interface IObservable<out T> {
IDisposable Subscribe(IObserver<T> observer);
}
// Events stream consumers
public interface IObserver<in T> {
void OnNext(T value);
void OnError(Exception error);
void OnCompleted();
}
}
Event processing at all scales with Reactive Extensions Bart de SmetThrusday 17 april 13:15 - 14:30
DemoUsing TX with LINQPad
Event Sources
• Tx.Windows• Event Tracing for Windows (ETW)• Performance Counters - interop with the "Performance Data Helper"
(PDH) native API• Event Logs (.evtx)• IIS W3C text log files
• Tx.SqlServer• SQL Server Extended Events (XEvent) is the tracing technology used by
the SQL-Server engine.
• Extensibility
Type Generation
Playback
•Structured Mode• Lose analogy is that events of given type are like a "Table“• playback.GetObservable< T> is similar to dataContext.GetTable<T> in
LINQ to SQL• More precisely, IObservable is a sequence (i.e. order matters) • This is the same distinction as IEnumerable vs.. SQL Table
•Timeline Mode• This mode is useful to ask questions that are about the timeline of all
events, regardless of type
Playback Features
•Multiplexed sequences
•Hiding the heterogeneity of the event sources
•Same API for real-time and past history
•Single-Read to answer multiple queries from file(s)
•Occurrence Time Scheduler
DemoHttpSpew
Case Study XblWatch
•Objective of XblWatch:• Replace existing rule based SCOM pack monitoring of web service• Provide near-realtime monitoring• Create Synthetic Performance Counters for success and failure rates
per API and service• Publish Synthetic Performance Counters to integrate with existing
monitoring and alerting system
XblWatch/erxblwatcher.prod.live
XblWatcher.exe XblWatchCmd.exehttp://*:80
tcp://*:9000
Perf Counters Site/Service/API
Machine 1
XblWatch.exe
http.sys ETW trace
Perf CountersMachine/Service/API
Machine 2
XblWatch.exe
http.sys ETW trace
Perf CountersMachine/Service/API
Machine 3
XblWatch.exe
http.sys ETW trace
Perf CounterMachine/Service/API
Machine N
XblWatch.exe
http.sys ETW trace
Perf CountersMachine/Service/API
DemoXblWatch
Resources
•Tx (LINQ to Logs and Traces)
• (Rx) Reactive Extensions
•Event processing at all scales with Reactive Extensions Bart de SmetThrusday 17 april 13:15 - 14:30
Laat ons weten wat u vindt van deze sessie! Vul de evaluatie in via www.techdaysapp.nl en maak kans op een van de 20 prijzen*. Prijswinnaars worden bekend gemaakt via Twitter (#TechDaysNL). Gebruik hiervoor de code op uw badge.
Let us know how you feel about this session! Give your feedback via www.techdaysapp.nl and possibly win one of the 20 prices*. Winners will be announced via Twitter (#TechDaysNL). Use your personal code on your badge.
* Over de uitslag kan niet worden gecorrespondeerd, prijzen zijn voorbeelden – All results are final, prices are examples
