linq, take two realizing the linq to everything dream bart j.f. de smet senior software development...

LINQ, Take TwoRealizing the LINQ to Everything Dream

Bart J.F. De SmetSenior Software Development EngineerMicrosoft Corporation

A Historical Perspective

Censored

7 years ago

Little recent innovation

Where’s the cloud?

Why?

What?How?

Language Integrated QueryMonads

IEnumerable<T>

IQueryable<T>

new[]{ 42 }

SelectMany

IEnumerable<R> SelectMany<T, R>( this IEnumerable<T> source, Func<T, IEnumerable<R>> selector)

Could there be more?

Also see www.codeplex.com/LINQSQO for “Project MinLINQ”

The Monadic Bind Operator Revealed

from _ in name from s in _.ToUpper() select s

name.SelectMany( _ => _.ToUpper(), s => s)Compiler

Null-propagating dot

string s = name?.ToUpper();

Syntactic sugar

One single library function suffices

Building the Maybe MonadFor Fun and No (?) Profit

demo

Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team

Essential LINQ

var src = new Source<Product>();

var res = from p in src where p.Price > 100m group p by p.Category;

foreach (var p in res) Console.WriteLine(p);

ImplementsIQueryable<T

>

Compiles fine

Does it really have to be a runtime check?

Query Providers RevisitedDo We Need IQueryable<T>?

var res = from p in src

where p.Price > 100m

group p by p.Category;

var res = src

.Where(p => p.Price > 100m)

.GroupBy(p => p.Category);

Syntactic sugar

Can be instance methods

Source<T> Filtered<T> Projected<T>Where Select

No GroupBy “edge”

Leveraging The Query Pattern

var res = from tweet in twitter where tweet.

AboutFromLocation

== “Bart”From

From

AboutLocation

== “LINQ”About

where tweet. About

Query “learns”

class Twitter{ public TwitterByFrom Where(Func<TweetAboutFromLoc, FilterFrom> filter);

// Other filter methods}

Recipe

Method overloadsType state machineOperator overloads

“Has a” type

From operator overloadingclass TwitterByFrom

{ public TwitterByAboutFrom Where(Func<TweetAboutLoc, FilterAbout> filter);

// Other filter methods // Fields with current filters}

select tweet;

Custom syntax trees

Taking It One Step Further

class TweetAboutFromLoc{ public FromString From; public AboutString About; public LocString Location;}

class FromString{ FilterFrom operator ==( FromString f, string s)}

class TweetAboutLoc{ public AboutString About; public LocString Location;}

class AboutString{ FilterAbout operator ==( AboutString f, string s)}

demo

Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team

Query Providers Revisited

Bottom-Up Tree RewritingA rule drive query provider

+

1*

2 3

Source domain

Rule Set

Patterns with rewrite actions

Target domain

BURS Tree Rewriting Engine

E.g. LINQ Expression Trees E.g. T-SQL Expression Trees

Const(2) Const(3)

Mul

Inc

Currently built at runtime, e.g. once per LINQ provider

that gets loaded

x1 $

Const(x)

State TablesLeaf node

? : int 1

Final states

1 1 $

Any source leaf node with an int constant value can be

reduced using rule 1.

Final states have a cost (incremental, see further).

x

+

a b

1 $Const(x)

2 $Add(a,b)

State TablesLeaf node

? : int 1

Wildcard

a : int 2 *

+ 2 *

2 * 3

Final states

1 1 $

3 2 $

Wildcard b doesn’t get its own entry, since it has the same type as wildcard a.

This reduces space in the BURS tables.

2D table for state transition of the +

operator.

Wildcard matches any subtree that’s assignable to int.

x

+

a b

*

a b

1 $Const(x)

2 $Add(a,b)

3 $Mul(a,b)

State TablesLeaf node

? : int 1

Wildcard

a : int 2 *

+ 2 *

2 * 3

* 2 *

2 * 4

Final states

1 1 $

3 2 $

4 3 $

What’s the meaning of costs?

The cost of 3$ is incremental and has to be added to the cost of all the wildcard matches.

In this case, a match of the Mul-rule (4 ) costs 3$ + costa + costb.

x

+

a b

*

a b

+

c*

a b

1 $Const(x)

2 $Add(a,b)

3 $Mul(a,b)

4 $AddMul(a,b,c)

State TablesLeaf node

? : int 1

Wildcard

a : int 2 *

+ 2 *

2 * 3

4 5

* 2 *

2 * 4

Final states

1 1 $

3 2 $

4 3 $

5 4 $

A pattern forest is kept to map

subtrees on already existent states. Here (a * b) matches state 4.

x

+

a b

*

a b

+

c*

a b

+

a 1

1 $Const(x)

2 $Add(a,b)

3 $Mul(a,b)

4 $AddMul(a,b,c)

1 $Inc(a)

State TablesLeaf node

? : int 1

1 6

Wildcard

a : int 2 *

+ 2 * 6

2 * 3 7

4 5

* 2 *

2 * 4

Final states

1 1 $

3 2 $

4 3 $

5 4 $

7 1 $

Subject Tree Coverings

2 3

* 1

+

Leaf node

? : int 1

1 6

Wildcard

a : int 2 *

+ 2 * 6

2 * 3 7

4 5

* 2 *

2 * 4

Final states

1 1 $

3 2 $

4 3 $

5 4 $

7 1 $

1 $ 1 $

5 $

6 $Constant leaf node 1

doesn’t contribute cost (part of pattern).

demo

Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team

Bottom Up Rewriting

Environment

MoveN

ext

Got next?

Application

On

Next

Have next!

IEnumerable<T>IEnumerator<T>

IObservable<T>IObserver<T>

Inte

racti

ve R

eactiv

eData Retrieval – Push or Pull?

Event Streams

Towards a unified programming modelProducers are observable sequences

.NET events, WinRT events, sensor APIs, APM methods, tasks, etc.

Consumers are observersHooking up “continuations” or handlers

Observable

Subscribe

Observer

Event Stream Processing Interfacesnamespace System{ public interface IObservable<out T> { IDisposable Subscribe(IObserver<T> observer); }

public interface IObserver<in T> { void OnNext(T value); void OnError(Exception error); void OnCompleted(); }}

Fix

ed

(MS

IL)

Tran

sla

tab

le

(Exp

ressio

n

trees)

ToQueryable

ToObservable

ToEnumerable

AsQ

uery

ab

le

AsEn

um

era

ble

AsQ

bserv

ab

le

AsO

bserv

ab

le

Pull(interactive)

Push(reactive)

Concurre

ncy

(ISch

eduler)

LINQ to *.*

What?

Where

?

How

?

Worker pools Message loopsThreads Distributed

Duality

Hom

o-

icon

ic

IQbservable<T>LINQ to WMI Events

ToQbservableIQueryable<T>

LINQ to SQL

IEnumerable<T>LINQ to Objects

IObservable<T>LINQ to Events

IQbservable<T> - The Dual of IQueryable<T>public interface IQbservable<out T> : IObservable<T>{ Expression Expression { get; } Type ElementType { get; } IQbservableProvider Provider { get; }}

public interface IQbservableProvider{ IQbservable<R> CreateQuery<R>(Expression expression);}

Extended role for some operators

No Execute method

Homoiconic naming

IQbservableIObservable

demo

Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team

LINQ to Twitter

Baked in notion of “where”?

foreach (var p in res.RemoteOn(new SqlScheduler(“server”))) // Process product

Decoupled “what” from “where”

Entry-point for the schema

Custom schedulers could be very rich (e.g. server farm)

The IScheduler abstractionWhere things happen

var ctx = new NorthwindDataContext(); var res = from product in ctx.Products where product.Price > 100m select product.Name;

Expression Tree Remoting

Rx .NET

RxJSstocks.Where(function (t) { return t.Symbol == “MSFT”; }).Select(function (t) { return t.Quote; })

JSON serializer

Observable data source

Retargeting to AJAX

from ticker in stockswhere ticker.Symbol == “MSFT”select ticker.Quote

Query Server

Alice F9F10

chat

talk

var chat = svc.GetStream<Message>(“chat”);

var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”);

talk.Subscribe(s => ShowMessage(s)); F5

chatchatchat

Where

Select

=

msg “Bob”

From

msg

Text

Translate

Translate

chat

Where

Select

=

msg “Bob”

From

msg

Text

Query Server

Alice

chat

talk

var chat = svc.GetStream<Message>(“chat”);

var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”);

talk.Subscribe(s => ShowMessage(s));

Bob F9

chat

F10

var chat = svc.GetStream(“chat”);

Rx.Observable.FromDOMEvent(txt, “Changed”) .Select(function (e) { return txt.Text; }) .Throttle(.5 /* 500 ms */) .Subscribe(function (s) { chat.OnNext(new Message(“Bob”, s)); }); F5

chat

Where

Select

=

msg “Bob”

From

msg

Text

Translate

chat

chat

Query Server

Alice

chat

talk

Bob

chat

Hello Alice! F5

var chat = svc.GetStream(“chat”);

Rx.Observable.FromDOMEvent(txt, “Changed”) .Select(function (e) { return txt.Text; }) .Throttle(.5 /* 500 ms */) .Subscribe(function (s) { chat.OnNext(new Message(“Bob”, s)); });

Bob says:

Hello Alice!

Hello Alice!

Bonjour Alice!

var chat = svc.GetStream<Message>(“chat”);

var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”);

talk.Subscribe(s => ShowMessage(s)); F5

Bob says:

Bonjour Alice!

Where

Select

=

msg “Bob”

From

msg

Text

Translate

“Bonjour Alice!”

From: “Bob”Text: “Hello Alice!”

From: “Bob”Text: “Hello Alice!”

Translate“Bonjour Alice!”

“Hello Alice!”

demo

Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team

Expression Tree Serialization

announcing

Now with support for hosting expression tree based queries (enumerable and observable)

StreamInsight v2.1

LINQ to the UnexpectedModel[ Decisions[Reals, SA, VZ], Goals[ Minimize[20 * SA + 15 * VZ] ], Constraints[ C1 -> 0.3 * SA + 0.4 * VZ >= 2000, C2 -> 0.4 * SA + 0.2 * VZ >= 1500, C3 -> 0.2 * SA + 0.3 * VZ >= 500, C4 -> SA <= 9000, C5 -> VZ <= 6000, C6 -> SA >= 0, C7 -> VZ >= 0 ]]

from m in ctx.CreateModel(new { vz = default(double), sa = default(double)})where 0.3 * m.sa + 0.4 * m.vz >= 2000 && 0.4 * m.sa + 0.2 * m.vz >= 1500 && 0.2 * m.sa + 0.3 * m.vz >= 500where 0 <= m.sa && m.sa <= 9000 && 0 <= m.vz && m.vz <= 6000orderby 20 * m.sa + 15 * m.vzselect m

To compute call Solve

Symbolic, non-persisted

Cost / barrel

Max# barrels

Min# barrels

Refin

em

en

t %

LINQ to the Unexpected

from costSA in GetPriceMonitor("SA")from costVZ in GetPriceMonitor("VZ")

from m in ctx.CreateModel(new { vz = default(double), sa = default(double) })

where 0.3 * m.sa + 0.4 * m.vz >= 2000 && 0.4 * m.sa + 0.2 * m.vz >= 1500 && 0.2 * m.sa + 0.3 * m.vz >= 500where 0 <= m.sa && m.sa <= 9000 && 0 <= m.vz && m.vz <= 6000

orderby costSA * m.sa + costVZ * m.vzselect m

Observabledata sources

Sele

ctM

an

y

Parametersto decision

Subscribe here!

demo

Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team

LINQ to Constraints

Related Content

DEV413 – Curing your Event Processing Blues using Reactive Extensions (Rx)

Find Me Later The Week At The Ask The Experts

Resources

Connect. Share. Discuss.

http://europe.msteched.com

Learning

Microsoft Certification & Training Resources

www.microsoft.com/learning

TechNet

Resources for IT Professionals

http://microsoft.com/technet

Resources for Developers

http://microsoft.com/msdn

Evaluations

http://europe.msteched.com/sessions

Submit your evals online

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS

PRESENTATION.