class 19 auto-tuning database kernelsdaslab.seas.harvard.edu › classes › cs165 › doc ›...

auto-tuning database kernelsprof. Stratos Idreos

HTTP://DASLAB.SEAS.HARVARD.EDU/CLASSES/CS165/

class 19

http://daslab.seas.harvard.edu/classes/cs165/

Stratos Idreos

now you know the basics!

and it is time for research

next: design a full adaptive data system kernel from scratch

Stratos Idreos

soon everyone will need to be a “data scientist”hmm,

my data is too big :(

1

new applications/requirements

how far away are we from a future where a data system sits in the critical path of everything we do?

Stratos’ theory #1

Stratos Idreos

years

daily

dat

a

[IBMbigdata]years

daily

dat

a

[Stratos’Guess]

data

* ski

llsdata system design,

set-up, tune, use

2Stratos’ theory #2

Stratos Idreos

design & build

data systems that are easy to:

(years)

set-up & tune

(months)

use

(hours/days)

Stratos Idreos

+ of course make things fast

Stratos Idreos

not always sure what we are looking for (until we find it)

data exploration

data has always been bigvolume velocity variety veracity

3

Stratos Idreos

today and Wednesday

adaptive indexing,adaptive loading, adaptive storage,dbTouch+ more

since 2005

Martin Kersten Stefan Manegold Goetz Graefe Harumi Kuno Anastasia Ailamaki Eleni Petraki

Ioannis Alagiannis Miguel Branco Renata Borovica Felix Halim Ronald Yap Panos Karras

joint work with:

Stratos Idreos

timeline

storage indexing query

expert users - idle time - workload knowledge

too many preparation options lead to complex installation

schema load

Stratos Idreos

users/applicationsdeclarative interface

ask what you want

db system

DBA

Stratos Idreos

users/applications

data system 1 data system 2

…

need to choose the proper system

& workloads/

applications change rapidly

DBA2DBA1

Stratos Idreos

storage indexing queryschema loadX X X

be able to query the data immediately & with good performance

Stratos Idreos

storage indexing queryschema loadX X X

be able to query the data immediately & with good performance

raw data explore data and gain knowledge “immediately”

Stratos Idreos

tune= create proper indices offlineperformance 10-100X

load indexing query

indexingstorage

Stratos Idreos

tune= create proper indices offlineperformance 10-100X

but it depends on workload!which indices to build? on which data parts? and when to build them?

load indexing query

indexingstorage

Stratos Idreos

queryload indexingstorage

Stratos Idreos

timeline


Stratos Idreos

timeline

sample workload


Stratos Idreos

timeline

sample workload analyze


Stratos Idreos

timeline

sample workload analyze create indices


Stratos Idreos

timeline

sample workload analyze create indices query


Stratos Idreos

timeline


complex and time consuming process


Stratos Idreos

timeline


complex and time consuming process

queryload indexingstoragehuman administrators + auto-tuning tools

Stratos Idreos

what can go wrong?

not enough idle time to finish proper tuning

by the time we finish tuning, the workload changes

big data V’s volume velocity variety veracity

not enough space to index all data

not enough money - energy - resources

Stratos Idreos

database cracking

Stratos Idreos

database cracking

idle timeworkload knowledge

external tools

human control

Stratos Idreos

database crackingauto-tuning database kernels

incremental, adaptive, partial indexing


external tools

human control

Stratos Idreos



indexing

initialization querying


external tools

human control

Stratos Idreos



indexing

initialization querying


external tools

human control

every query is treated as an advice on how data should be stored

Stratos Idreos

A B C D

...

...

relation1/table1

column-store databasea fixed-width and dense array per attribute

Database Cracking CIDR 2007

Stratos Idreos

13 16 4 9 2

12 7 1

19 3

14 11 8 6

column AQ1: select R.A from R where R.A>10 and R.A<14


Stratos Idreos

13 16 4 9 2

12 7 1

19 3

14 11 8 6


sort

1 2 3 4 6 7 8 9

11 12 13 14 16 19


Stratos Idreos

13 16 4 9 2

12 7 1

19 3

14 11 8 6


sort

1 2 3 4 6 7 8 9

11 12 13 14 16 19

binary search


Stratos Idreos

13 16 4 9 2

12 7 1

19 3

14 11 8 6


sort

1 2 3 4 6 7 8 9

11 12 13 14 16 19

binary search re

sult


Stratos Idreos

13 16 4 9 2

12 7 1

19 3

14 11 8 6


sort

1 2 3 4 6 7 8 9

11 12 13 14 16 19

binary search re

sult

time+

knowledge


Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6



Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10


Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14


Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14

piece3: A>=14


Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14

piece3: A>=14

resu

lt


Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14

piece3: A>=14

resu

lt

gain knowledge on how data is organized


Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14

piece3: A>=14

dynamically/on-the-fly within the select-operator

resu

lt

gain knowledge on how data is organized


Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14

piece3: A>=14

Q2: select R.A from R where R.A>7 and R.A<=16

dynamically/on-the-fly within the select-operatorDatabase Cracking CIDR 2007

Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14

piece3: A>=14

4 2 1 3 6 7 9 8 13 12 11 14 16 19

piece1: A<=7

piece2: 7<A<=10Q2: select R.A from R where R.A>7 and R.A<=16


Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14

piece3: A>=14

4 2 1 3 6 7 9 8 13 12 11 14 16 19

piece1: A<=7

piece2: 7<A<=10

piece3: 10<A<14Q2: select R.A from R where R.A>7 and R.A<=16


Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14

piece3: A>=14

4 2 1 3 6 7 9 8 13 12 11 14 16 19

piece1: A<=7

piece2: 7<A<=10

piece3: 10<A<14

piece4: 14<=A<=16

piece5: A>16



Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14

piece3: A>=14

4 2 1 3 6 7 9 8 13 12 11 14 16 19

piece1: A<=7

piece2: 7<A<=10

piece3: 10<A<14

piece4: 14<=A<=16

piece5: A>16



resu

lt


Stratos Idreos

13 16 4 9 2 12 7 1 19 3 14 11 8 6


4 9 2 7 1 3 8 6 13 12 11 16 19 14

piece1: A<=10

piece2: 10<A<14

piece3: A>=14

4 2 1 3 6 7 9 8 13 12 11 14 16 19

piece1: A<=7

piece2: 7<A<=10

piece3: 10<A<14

piece4: 14<=A<=16

piece5: A>16



resu

lt

the more we crack, the more we learn


Stratos Idreos

select [15,55]


Stratos Idreos

10 20 30 40 50 60

select [15,55]

select [15,55]


Stratos Idreos

10 20 30 40 50 60

select [15,55]

select [15,55]pieces become smaller and smaller

touch at most two pieces at a time


Stratos Idreos

# of queries

resp

onse

tim

estandard

optimal

but requires time, workload knowledge & stability

Stratos Idreos

# of queries

resp

onse

tim

estandard

optimal

adaptive

but requires time, workload knowledge & stability

Stratos Idreos

100K random selections random selectivity random value ranges in a 10 million integer column

set-upScan

Full Index

Crack

0.001

0.01

0.1

1

10

100

1000

1 10 100 1000 10000 100000

Resp

onse

tim

e (

secs

)

Query sequence (x1000)

continuous adaptation


Stratos Idreos


almost no initialization overhead

set-upScan

Full Index

Crack

0.001

0.01

0.1

1

10

100

1000

1 10 100 1000 10000 100000

Resp

onse

tim

e (

secs

)




Stratos Idreos


almost no initialization overhead

continuous improvement

set-upScan

Full Index

Crack

0.001

0.01

0.1

1

10

100

1000

1 10 100 1000 10000 100000

Resp

onse

tim

e (

secs

)




Stratos Idreos

10K random selections selectivity 10% random value ranges in a 30 million integer column


set-up

0.004

200

0.001

0.01

0.1

1

10

100

1 10 100 1000 10000

Cum

ula

tive a

vera

ge r

esp

onse

tim

e (

secs

)

Query sequence

Scan

Full Index

Crack


Stratos Idreos

10K random selections selectivity 10% random value ranges in a 30 million integer column

10K queries later, Full Index still has not

amortized the initialization costs


set-up

0.004

200

0.001

0.01

0.1

1

10

100

1 10 100 1000 10000

Cum

ula

tive a

vera

ge r

esp

onse

tim

e (

secs

)

Query sequence

Scan

Full Index

Crack


Stratos Idreos

Atable1

Stratos Idreos

A B C D

...

...

table1

Stratos Idreos

A B C D

...

...

table1

select R.A from R where R.A>10 and R.A<14

Stratos Idreos

A B C D

...

...

table1

select R.A from R where R.A>10 and R.A<14select max(R.A),max(R.B),max(S.A),max(S.B) from R,S where v1 <R.C<v2 and v3 <R.D<v4 and v5 <R.E<v6 and k1 <S.C<k2 and k3 <S.D<k4 and k5 <S.E<k6 and R.F = S.F

Stratos Idreos

A B C D

...

...

table1

select R.A from R where R.A>10 and R.A<14select max(R.A),max(R.B),max(S.A),max(S.B) from R,S where v1 <R.C<v2 and v3 <R.D<v4 and v5 <R.E<v6 and k1 <S.C<k2 and k3 <S.D<k4 and k5 <S.E<k6 and R.F = S.F

updatesjoins

concurrency control

...

cracking databases

updates (SIGMOD07)

storage-restrictions (SIGMOD09)

benchmarking (TPCTC10)

concurrency control

(PVLDB12)

algorithms (PVLDB11)

basics (CIDR07)

multi-cores (SIGMOD15)

hadoop (Yale/Saarland)

b-trees (HP Labs)

>1 columns (SIGMOD09)

robustness (PVLDB12)

adaptive storage

(SIGMOD14)

time-series (SIGMOD14)

encryption (SIGMOD16)

Stratos Idreos

updates

Updates SIGMOD 2007

Stratos Idreos

cracking indices are auxiliary data structurescan be dropped any time

Stratos Idreos

column of 10M tuples, random queries 1000 random insertions every 1000 queries

crack (forget) scan

forgettingwhen updates arrive, drop the index

Stratos Idreos


queries (x1000) queries (x1000)

crack (forget) scan

we do not exploit past cracking


Stratos Idreos


crack (forget) scan



Stratos Idreos



crack (forget) scan



Stratos Idreos

log updates and apply on-line and on-demand during cracking

cracking updates

maintain

updates query

Stratos Idreos

pieces are ordered

values in a piece are not ordered

goal: minimize physical actions

arrays are dense3

column

pending inserts

pending deletes

1

9

17

12

16

25

22

29

33

39

31

10

20

30

Stratos Idreos

pieces are ordered

values in a piece are not ordered

goal: minimize physical actions

arrays are dense3

column

pending inserts

pending deletes

1

9

17

12

16

25

22

29

33

39

31

10

20

30

11

Stratos Idreos


merge all updates merge only qualifying updates

Stratos Idreos




gradual merging avoids high picks

Stratos Idreos




Stratos Idreos





Stratos Idreos

the ripplecolumn

pending updates

Stratos Idreos

the ripplecolumn

pending updates

quer

y re

sult

Stratos Idreos

the ripplecolumn

pending updates

quer

y re

sult

doing work not relevant for the current query

Stratos Idreos

the ripplecolumn

pending updates

quer

y re

sult


column

quer

y re

sult

Stratos Idreos

the ripplecolumn

pending updates

quer

y re

sult


column

quer

y re

sult

update only the hot tuples

Stratos Idreos

forget ripple

Stratos Idreos



forget ripple

maintains adaptive behavior

Stratos Idreos

sideways crackinghow to process >1 columns

Sideways Cracking, SIGMOD 09

Stratos Idreos

column-stores

a1a2a3a4a5a6

b1b2b3b4b5b6

c1c2c3c4c5c6

positional alignment

positional lookupsA(i) = A + i * width(A)

Tuple 1Tuple 2Tuple 3Tuple 4Tuple 5Tuple 6

A B C


Stratos Idreos

select min(C) from R where A<10 & B<20

B<20 minCA<10A B C D IDs B CIDsdisk memory


Stratos Idreos

pos1 pos2

1 0 1 0

A

sort

edselect max(D),min(E) from R where (A>10 and A<40) and (B>20 and B<60)

binary search for 10 & 40

B

for all B values between pos1 & 2: if B>20 and B<60 mark bit vector at pos i

maxD

D

for each marked position max(D)

…

avoid scan of A avoid TR on B work on a restricted area across all columns good for memory hierarchy

Stratos Idreos

a1 a2 a3 a4 a5 a6

1 3 6

RowIDs

a1 a2 a3 a4 a5 a6

6 1 3

RowIDs

ordered

unordered

sequential access

random access

without cracking

with cracking


Stratos Idreos

A

sideways cracking


Stratos Idreos

A B C D

sideways cracking


Stratos Idreos

A B C Dquery sideways cracking


Stratos Idreos

A B C D

query

sideways cracking


Stratos Idreos

A B C D

log crack actions and replay to align columns dynamically

sideways cracking

replace tuple reconstruction with cracking actions Sideways Cracking, SIGMOD 09

Stratos Idreos

sideways cracking

12

3

5

9

15

22

7

26

4

2

24

11

16

b1

b2

b3

b4

b5

b6

b7

b8

b9

b10

b11

b12

b13

A B

select B from R where 10<A<15

4

3

5

9

2

7

12

11

15

22

24

26

16

b9

b2

b3

b4

b10

b7

b1

b12

b5

b6

b11

b8

b13

AB

Position 1

value <=10Piece 1

Position 7

value >10Piece 2

Position 9

value >=15Piece 3

select B from R where 5=<A<17

4

3

2

9

5

7

12

11

15

16

24

26

22

b9

b2

b10

b4

b3

b7

b1

b12

b5

b13

b11

b8

b6

Position 1

value <5Piece 1

Position 7

value >10Piece 3

Position 9

value >=15Piece 4

Position 4

value >=5Piece 2

Position 11

value >=17Piece 5

Cracker index Cracker indexM ABMInitial state


Stratos Idreos

sideways cracking

12

3

5

9

15

22

7

26

4

2

24

11

16

b1

b2

b3

b4

b5

b6

b7

b8

b9

b10

b11

b12

b13

A B


4

3

5

9

2

7

12

11

15

22

24

26

16

b9

b2

b3

b4

b10

b7

b1

b12

b5

b6

b11

b8

b13

AB

Position 1

value <=10Piece 1

Position 7

value >10Piece 2

Position 9

value >=15Piece 3


4

3

2

9

5

7

12

11

15

16

24

26

22

b9

b2

b10

b4

b3

b7

b1

b12

b5

b13

b11

b8

b6

Position 1

value <5Piece 1

Position 7

value >10Piece 3

Position 9

value >=15Piece 4

Position 4

value >=5Piece 2

Position 11

value >=17Piece 5


Cracker MapHead Tail


Stratos Idreos

sideways cracking

12

3

5

9

15

22

7

26

4

2

24

11

16

b1

b2

b3

b4

b5

b6

b7

b8

b9

b10

b11

b12

b13

A B


4

3

5

9

2

7

12

11

15

22

24

26

16

b9

b2

b3

b4

b10

b7

b1

b12

b5

b6

b11

b8

b13

AB

Position 1

value <=10Piece 1

Position 7

value >10Piece 2

Position 9

value >=15Piece 3


4

3

2

9

5

7

12

11

15

16

24

26

22

b9

b2

b10

b4

b3

b7

b1

b12

b5

b13

b11

b8

b6

Position 1

value <5Piece 1

Position 7

value >10Piece 3

Position 9

value >=15Piece 4

Position 4

value >=5Piece 2

Position 11

value >=17Piece 5



crack based on head, carry tail


Stratos Idreos

sideways cracking

12

3

5

9

15

22

7

26

4

2

24

11

16

b1

b2

b3

b4

b5

b6

b7

b8

b9

b10

b11

b12

b13

A B


4

3

5

9

2

7

12

11

15

22

24

26

16

b9

b2

b3

b4

b10

b7

b1

b12

b5

b6

b11

b8

b13

AB

Position 1

value <=10Piece 1

Position 7

value >10Piece 2

Position 9

value >=15Piece 3


4

3

2

9

5

7

12

11

15

16

24

26

22

b9

b2

b10

b4

b3

b7

b1

b12

b5

b13

b11

b8

b6

Position 1

value <5Piece 1

Position 7

value >10Piece 3

Position 9

value >=15Piece 4

Position 4

value >=5Piece 2

Position 11

value >=17Piece 5


Cracking knowledge




Stratos Idreos

sideways cracking

12

3

5

9

15

22

7

26

4

2

24

11

16

b1

b2

b3

b4

b5

b6

b7

b8

b9

b10

b11

b12

b13

A B


4

3

5

9

2

7

12

11

15

22

24

26

16

b9

b2

b3

b4

b10

b7

b1

b12

b5

b6

b11

b8

b13

AB

Position 1

value <=10Piece 1

Position 7

value >10Piece 2

Position 9

value >=15Piece 3


4

3

2

9

5

7

12

11

15

16

24

26

22

b9

b2

b10

b4

b3

b7

b1

b12

b5

b13

b11

b8

b6

Position 1

value <5Piece 1

Position 7

value >10Piece 3

Position 9

value >=15Piece 4

Position 4

value >=5Piece 2

Position 11

value >=17Piece 5


Cracking knowledge No tuple reconstruction




Stratos Idreos

sideways cracking

12

3

5

9

15

22

7

26

4

2

24

11

16

b1

b2

b3

b4

b5

b6

b7

b8

b9

b10

b11

b12

b13

A B


4

3

5

9

2

7

12

11

15

22

24

26

16

b9

b2

b3

b4

b10

b7

b1

b12

b5

b6

b11

b8

b13

AB

Position 1

value <=10Piece 1

Position 7

value >10Piece 2

Position 9

value >=15Piece 3


4

3

2

9

5

7

12

11

15

16

24

26

22

b9

b2

b10

b4

b3

b7

b1

b12

b5

b13

b11

b8

b6

Position 1

value <5Piece 1

Position 7

value >10Piece 3

Position 9

value >=15Piece 4

Position 4

value >=5Piece 2

Position 11

value >=17Piece 5


Dynamically/on-the-fly within the select-operatorCracking knowledge No tuple reconstruction




Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C

select B from R where A<3 select C from R where A<5 select B,C from R where A<4

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3

Crack A<3 Crack A<4Crack A<5

Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment

perform the same cracks and in the same order on all maps with the same head

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment


on-line alignment touch/load everything, always

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment


on-line alignment Xtouch/load everything, always

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment


remember and replay cracks across columns

on-line alignment Xtouch/load everything, always

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment

Align

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment

Align

Align

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment

Align

AlignCorrect

alignment

adaptive alignment


Stratos Idreos

7

4

1

2

8

3

6

B

b1

b2

b3

b4

b5

b6

b7

A

c1

c2

c3

c4

c5

c6

c7

C


2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

3

4

1

2

8

7

6

c6

c2

c3

c4

c5

c1

c7

v<5

v>=5

2

1

3

7

8

4

6

M

b4

b3

b6

b1

b5

b2

b7

v<3

v>=4

v>=3

3

2

1

4

8

7

6

c6

c4

c3

c2

c5

c1

c7

v<4

v>=5

v>=4

Result

b4

b3

Result

c6

c2

c3

c4

Result

b4

b3

b6

c6

c4

c3

Initial state

Without Alignment

2

1

4

7

8

3

6

b4

b3

b2

b1

b5

b6

b7

v<3

v>=3

b4

b3

Result

WithAlignment

2

1

4

3

8

7

6

c4

c3

c2

c6

c5

c1

c7

v<3

v>=5

Result

c4

c3

c2

c6

2

1

4

7

8

3

6

c4

c3

c2

c1

c5

c6

c7

v<3

v>=3

Crack A<5

v>=3

2

1

4

3

8

7

6

b4

b3

b2

b6

b5

b1

b7

v<3

v>=5

v>=3

2

1

3

4

8

7

6

b4

b3

b6

b2

b5

b1

b7

v<3

v>=5

v>=3

v>=4

2

1

3

4

8

7

6

c4

c3

c6

c2

c5

c1

c7

v<3

v>=5

v>=3

v>=4

Result

b4

b3

b6

c4

c3

c6

Wrong

allignment

Correct

allignment

Crack A<3

Crack A<5Crack A<3


Crack A<4

AB

MAC

MAB

MAC

MAC MAC MAC

MABMABMAB

Wrong alignment

Align

AlignCorrect

alignment

replace tuple reconstruction with cracking

adaptive alignment


Stratos Idreos

multi-selections

select D from R where 3<A<10 and 4<B<8 and 1<C<7

wider maps...but too many combinations maps of different maps sets lead to alignment problems


Stratos Idreos

multi-selections

use a single map set and exploit bit-vectors


wider maps...but too many combinations maps of different maps sets lead to alignment problems


Stratos Idreos

multi-selections

12

3

5

9

8

22

7

26

4

2

7

9

2

6

10

7

11

16

2

5

8

3

A B

select_create_bv(A,3,10,B,4,8)

2

3

5

9

8

7

7

4

26

12

22

8

2

6

10

7

3

16

5

2

9

11

MAB

v<=3

v>3

v>=10

1

0

1

0

0

1

Bit

vector bv

3

6

2

1

6

9

12

2

11

17

3

C

select_refine_bv(A,3,10,C,1,7,bv)

2

3

5

9

8

7

7

4

26

12

22

17

6

2

1

6

3

12

11

2

3

9

v<=3

v>3

v>=10

1

0

1

0

0

0

Bit

vector bv

9

4

2

10

12

19

3

6

5

8

1

D

reconstruct(A,3,10,D,bv)

2

3

5

9

8

7

7

4

26

12

22

8

4

2

10

12

1

3

5

6

9

19

v<=3

v>3

v>=10

2

12

Result

MAC MAD

select D from R

where 3<A<10 and

4<B<8 and 1<C<7

Initial stateQuery

1

0

1

0

0

0

Bit

vector bv



Stratos Idreos

multi-selections

12

3

5

9

8

22

7

26

4

2

7

9

2

6

10

7

11

16

2

5

8

3

A B


2

3

5

9

8

7

7

4

26

12

22

8

2

6

10

7

3

16

5

2

9

11

MAB

v<=3

v>3

v>=10

1

0

1

0

0

1

Bit

vector bv

3

6

2

1

6

9

12

2

11

17

3

C


2

3

5

9

8

7

7

4

26

12

22

17

6

2

1

6

3

12

11

2

3

9

v<=3

v>3

v>=10

1

0

1

0

0

0

Bit

vector bv

9

4

2

10

12

19

3

6

5

8

1

D


2

3

5

9

8

7

7

4

26

12

22

8

4

2

10

12

1

3

5

6

9

19

v<=3

v>3

v>=10

2

12

Result

MAC MAD

select D from R

where 3<A<10 and

4<B<8 and 1<C<7

Initial stateQuery

1

0

1

0

0

0

Bit

vector bv


Crack 3<A<10 Analyze tail 4<B<8 Create bit vector


Stratos Idreos

multi-selections

12

3

5

9

8

22

7

26

4

2

7

9

2

6

10

7

11

16

2

5

8

3

A B


2

3

5

9

8

7

7

4

26

12

22

8

2

6

10

7

3

16

5

2

9

11

MAB

v<=3

v>3

v>=10

1

0

1

0

0

1

Bit

vector bv

3

6

2

1

6

9

12

2

11

17

3

C


2

3

5

9

8

7

7

4

26

12

22

17

6

2

1

6

3

12

11

2

3

9

v<=3

v>3

v>=10

1

0

1

0

0

0

Bit

vector bv

9

4

2

10

12

19

3

6

5

8

1

D


2

3

5

9

8

7

7

4

26

12

22

8

4

2

10

12

1

3

5

6

9

19

v<=3

v>3

v>=10

2

12

Result

MAC MAD

select D from R

where 3<A<10 and

4<B<8 and 1<C<7

Initial stateQuery

1

0

1

0

0

0

Bit

vector bv



Align 3<A<10 Analyze tail 1<C<7 Refine bit vector


Stratos Idreos

multi-selections

12

3

5

9

8

22

7

26

4

2

7

9

2

6

10

7

11

16

2

5

8

3

A B


2

3

5

9

8

7

7

4

26

12

22

8

2

6

10

7

3

16

5

2

9

11

MAB

v<=3

v>3

v>=10

1

0

1

0

0

1

Bit

vector bv

3

6

2

1

6

9

12

2

11

17

3

C


2

3

5

9

8

7

7

4

26

12

22

17

6

2

1

6

3

12

11

2

3

9

v<=3

v>3

v>=10

1

0

1

0

0

0

Bit

vector bv

9

4

2

10

12

19

3

6

5

8

1

D


2

3

5

9

8

7

7

4

26

12

22

8

4

2

10

12

1

3

5

6

9

19

v<=3

v>3

v>=10

2

12

Result

MAC MAD

select D from R

where 3<A<10 and

4<B<8 and 1<C<7

Initial stateQuery

1

0

1

0

0

0

Bit

vector bv




Align 3<A<10 Grab tail values


Stratos Idreos

multi-selections

12

3

5

9

8

22

7

26

4

2

7

9

2

6

10

7

11

16

2

5

8

3

A B


2

3

5

9

8

7

7

4

26

12

22

8

2

6

10

7

3

16

5

2

9

11

MAB

v<=3

v>3

v>=10

1

0

1

0

0

1

Bit

vector bv

3

6

2

1

6

9

12

2

11

17

3

C


2

3

5

9

8

7

7

4

26

12

22

17

6

2

1

6

3

12

11

2

3

9

v<=3

v>3

v>=10

1

0

1

0

0

0

Bit

vector bv

9

4

2

10

12

19

3

6

5

8

1

D


2

3

5

9

8

7

7

4

26

12

22

8

4

2

10

12

1

3

5

6

9

19

v<=3

v>3

v>=10

2

12

Result

MAC MAD

select D from R

where 3<A<10 and

4<B<8 and 1<C<7

Initial stateQuery

1

0

1

0

0

0

Bit

vector bv




Align 3<A<10 Grab tail values

Use histogram-like info from maps to choose map set


Stratos Idreos

70

330

100

150

200

250

300

0 5 10 15 20 25 30Query sequence

TPC-H Query 15764420

1000

10000

Sel. Crack

Sid. Crack

MonetDB

PresortedMySQL

Presorted

Resp

onse

tim

e (m

illi s

ecs)


Stratos Idreos

70

330

100

150

200

250

300



1000

10000

Sel. Crack

Sid. Crack

MonetDB

PresortedMySQL

Presorted

Resp

onse

tim

e (m

illi s

ecs)

normal MonetDB

selection cracking


Stratos Idreos

70

330

100

150

200

250

300



1000

10000

Sel. Crack

Sid. Crack

MonetDB

PresortedMySQL

Presorted

preparation cost 3-14 minutes

per query

Resp

onse

tim

e (m

illi s

ecs)

presorted MonetDB

normal MonetDB

selection cracking


Stratos Idreos

70

330

100

150

200

250

300



1000

10000

Sel. Crack

Sid. Crack

MonetDB

PresortedMySQL

Presorted

preparation cost 3-14 minutes

per query

Resp

onse

tim

e (m

illi s

ecs)

presorted MonetDBMonetDB with

sideways cracking

normal MonetDB

selection cracking


Stratos Idreos

Partial Sideways Cracking

15

8

19

6

11

2

14

5

12

18

4

9

13

7

b1

b2

b3

b4

b5

b6

b7

b8

b9

b10

b11

b12

b13

b14

A B

Select B from R where 9<A<=15Initial state

Chunk map H (A,id)

v<=9

U

v>9

F

v>15

U

v<=9

E

v>9

M,C=1

v>15

E

Partial map M

Select B from R where 9<A<13

v<=9

E

v>9

M,C=1

v>15

E

v>=13

M,C=1

Select B from R where 5<=A<8

v<5

U

v>9

F

v>15

Uv>15, E

v>9

M,C=1

v>=13

M,C=1

v>=5

F

v>=8

U

v<5, E

v>=5

M,C=2

v>=8, E

7

8

9

6

4

2

5

14

12

13

15

11

19

18

14

2

12

4

11

6

8

7

9

13

1

5

3

10

14

12

13

15

11

b7

b9

b13

b1

b5

11

12

13

15

14

b5

b9

b13

b1

b7

4

2

7

6

5

9

8

14

12

13

15

11

19

18

11

6

14

11

8

12

2

7

9

13

1

5

3

10

11

12

13

15

14

b5

b9

b13

b1

b7

7

6

5

b14

b11

b8

c1

c2

c3

c4

c5

c6

c7

c8

c9

c10

c11

c12

c13

c14

C

Select C from R where 8<=A<15

v<5

U

v>9

F

v>15

U v>15, E

v>9

M,C=1

v>=13

M,C=1

v>=5

F

v>=8

F

v<8, E

v>=8

M,C=3

4

2

7

6

5

9

8

14

12

13

15

11

19

18

11

6

14

11

8

12

2

7

9

13

1

5

3

10

11

12

13

14

15

c5

c9

c13

c7

c1

9

8

c12

c2

v>15

M,C=1

A

Chunk map H (A,id)A Chunk map H (A,id)A

AB Partial map MAB

Partial map MAB Partial map MAC

1

2

3

4

5

6

7

8

9

10

11

12

13

14

Keys

Stratos Idreos

Stratos Idreos

crack

crack

Stratos Idreos

crack

crack

no alignment

no alignment

Stratos Idreos

crack

crack

full alignment

full alignment

no alignment

no alignment

Stratos Idreos

crack

crack

full alignment

full alignment

partial alignment

partial alignment

no alignment

no alignment

Stratos Idreos

concurrency control

Concurrency Control, PVLDB 12

problem: read queries become write queries! (?)

goal: be able to crack for multiple queries in parallel

Stratos Idreos

traditional indexing adaptive indexing


Stratos Idreos

traditional indexing adaptive indexingwrite queries read queries


Stratos Idreos


change index contents and structure

only index structure changes

write queries read queries


Stratos Idreos




no need for traditional locks = too heavy

short term latches = fast and release quickly



Stratos Idreos


all or nothing incremental and optional





Stratos Idreos





v1 v2 v3



Stratos Idreos





v1 v2 v3

a b



Stratos Idreos





v1 v2 v3 v1 v2 v3

a b

a b



Stratos Idreos





v1 v2 v3 v1 v2 v3

a b

a b

v1 v2 v3a



Stratos Idreos



impact stable storage stable storage optional





Stratos Idreos






v1 v2 v3 v1 v2 v3a b


a b


Stratos Idreos






need to serialize can execute in any order



Stratos Idreos

fewer conflicts as we adapt

query sequence

confl

icts

Scan

Full Index

Crack

0.001

0.01

0.1

1

10

100

1000

1 10 100 1000 10000 100000

Resp

onse

tim

e (

secs

)



Stratos Idreos

late reconstruction


diskA B C D


Stratos Idreos

late reconstruction


A<10disk memory

A B C D


Stratos Idreos

late reconstruction


A<10disk memory

A B C D IDs


Stratos Idreos

late reconstruction


A<10disk memory

A B C D IDs B


Stratos Idreos

late reconstruction


B<20A<10disk memory

A B C D IDs B


Stratos Idreos

late reconstruction


B<20A<10disk memory

A B C D IDs B CIDs


Stratos Idreos

late reconstruction


B<20 minCA<10disk memory

A B C D IDs B CIDs


Stratos Idreos

late reconstruction


B<20 minCA<10disk memory

A B C D IDs B CIDs

column lock and release as soon as an operator completes


Stratos Idreos Concurrency Control, PVLDB 12

select [a,b]

Stratos Idreos

need to latch only to be cracked pieces (max 2 per select)


select [a,b]

Stratos Idreos

piece locking

avl-tree crack column


Stratos Idreos

piece locking

avl-tree crack columncrack select


Stratos Idreos

piece locking

avl-tree crack columncrack selectwlock


Stratos Idreos

piece locking

avl-tree crack columncrack select

wlock


Stratos Idreos

piece locking

avl-tree crack column


Stratos Idreos

piece locking

avl-tree crack columnmax


Stratos Idreos

piece locking

avl-tree crack columnmaxrlock


Stratos Idreos

piece locking

avl-tree crack columnmax

rlock


Stratos Idreos

avl-tree crack column1090

140

200

300

piece locking

Stratos Idreos

avl-tree crack column crack select1090

140

200

300

65

230

piece locking

Stratos Idreos

avl-tree crack column crack selectwlock

wlock

1090

140

200

300

65

230

piece locking

Stratos Idreos


wlock

r/wlock

1090

140

200

300

65

230

piece locking

Stratos Idreos


wlock

q1,q2,q3...qn1090

140

200

300

65

230

10

90

piece locking

Stratos Idreos


wlock

q1,q2,q3...qn1090

140

200

300

65

230

10

90

2030

7080

piece locking

Stratos Idreos


wlock

q1,q2,q3...qn1090

140

200

300

65

230

10

90

2030

7080

wlock

piece locking

Stratos Idreos

0

10

20

30

40

50

60

70

80

1 2 4 8 16 32

Tota

l Tim

e for

1024 Q

ueries

(secs

)

Number of Concurrent Clients

scansort

crack

300

0

50

100

150

200

250

350

1 2 4 8 16 32

Thro

ughput ove

r 1024 Q

ueries

(Queries

/ se

c)

Number of Concurrent Clients

scansort

crack

4.5

4.55

4.6

4.65

4.7

4.75

4.8

Tota

l Tim

e for

1024 Q

ueries

(secs

)

(Sequential Execution)

Concurrency Control

Enabled Disabled

Rows: 100M Query: select sum(A) from R where v1 < A1 < v2 Selectivity: 0.01%, Random, # of queries: 1024 Clients: 1-32, Machine: 4 cores

adaptive indexing maintains its performance advantage


Stratos Idreos

crack time and wait time

10-6

10-5

10-4

10-3

10-2

10-1

100

1 10 100 1000

Tim

e p

er

query

(se

cs)

Query sequence

index refinementwait time

Query: select sum(A) from R where v1 < A1 < v2 Selectivity: 0.01%, Random, # of queries: 1024 Clients: 8, Machine: 4 cores

adaptive indexing maintains

the adaptive behavior

adaptive behavior also for conflicts


Stratos Idreos

Overview of Data Exploration TechniquesS. Idreos, O. Papaemmanouil, and S. Chaudhuri ACM SIGMOD International Conference on Management of Data, 2015

(explore citations)

DATA SYSTEMSprof. Stratos Idreos

class 19

auto-tuning database kernels

class 19 auto-tuning database kernelsdaslab.seas.harvard.edu › classes › cs165 › doc ›...

Documents