class 19 auto-tuning database kernelsdaslab.seas.harvard.edu › classes › cs165 › doc ›...
Post on 28-Jun-2020
2 Views
Preview:
TRANSCRIPT
auto-tuning database kernelsprof. Stratos Idreos
HTTP://DASLAB.SEAS.HARVARD.EDU/CLASSES/CS165/
class 19
Stratos Idreos
now you know the basics!
and it is time for research
next: design a full adaptive data system kernel from scratch
Stratos Idreos
soon everyone will need to be a “data scientist”hmm,
my data is too big :(
1
new applications/requirements
how far away are we from a future where a data system sits in the critical path of everything we do?
Stratos’ theory #1
Stratos Idreos
years
daily
dat
a
[IBMbigdata]years
daily
dat
a
[Stratos’Guess]
data
* ski
llsdata system design,
set-up, tune, use
2Stratos’ theory #2
Stratos Idreos
design & build
data systems that are easy to:
(years)
set-up & tune
(months)
use
(hours/days)
Stratos Idreos
+ of course make things fast
Stratos Idreos
not always sure what we are looking for (until we find it)
data exploration
data has always been bigvolume velocity variety veracity
3
Stratos Idreos
today and Wednesday
adaptive indexing,adaptive loading, adaptive storage,dbTouch+ more
since 2005
Martin Kersten Stefan Manegold Goetz Graefe Harumi Kuno Anastasia Ailamaki Eleni Petraki
Ioannis Alagiannis Miguel Branco Renata Borovica Felix Halim Ronald Yap Panos Karras
joint work with:
Stratos Idreos
timeline
storage indexing query
expert users - idle time - workload knowledge
too many preparation options lead to complex installation
schema load
Stratos Idreos
users/applicationsdeclarative interface
ask what you want
db system
DBA
Stratos Idreos
users/applications
data system 1 data system 2
…
need to choose the proper system
& workloads/
applications change rapidly
DBA2DBA1
Stratos Idreos
storage indexing queryschema loadX X X
be able to query the data immediately & with good performance
Stratos Idreos
storage indexing queryschema loadX X X
be able to query the data immediately & with good performance
raw data explore data and gain knowledge “immediately”
Stratos Idreos
tune= create proper indices offlineperformance 10-100X
load indexing query
indexingstorage
Stratos Idreos
tune= create proper indices offlineperformance 10-100X
but it depends on workload!which indices to build? on which data parts? and when to build them?
load indexing query
indexingstorage
Stratos Idreos
queryload indexingstorage
Stratos Idreos
timeline
queryload indexingstorage
Stratos Idreos
timeline
sample workload
queryload indexingstorage
Stratos Idreos
timeline
sample workload analyze
queryload indexingstorage
Stratos Idreos
timeline
sample workload analyze create indices
queryload indexingstorage
Stratos Idreos
timeline
sample workload analyze create indices query
queryload indexingstorage
Stratos Idreos
timeline
sample workload analyze create indices query
complex and time consuming process
queryload indexingstorage
Stratos Idreos
timeline
sample workload analyze create indices query
complex and time consuming process
queryload indexingstoragehuman administrators + auto-tuning tools
Stratos Idreos
what can go wrong?
not enough idle time to finish proper tuning
by the time we finish tuning, the workload changes
big data V’s volume velocity variety veracity
not enough space to index all data
not enough money - energy - resources
Stratos Idreos
what can go wrong?
not enough idle time to finish proper tuning
by the time we finish tuning, the workload changes
big data V’s volume velocity variety veracity
not enough space to index all data
not enough money - energy - resources
Stratos Idreos
database cracking
Stratos Idreos
database cracking
idle timeworkload knowledge
external tools
human control
Stratos Idreos
database crackingauto-tuning database kernels
incremental, adaptive, partial indexing
idle timeworkload knowledge
external tools
human control
Stratos Idreos
database crackingauto-tuning database kernels
incremental, adaptive, partial indexing
indexing
initialization querying
idle timeworkload knowledge
external tools
human control
Stratos Idreos
database crackingauto-tuning database kernels
incremental, adaptive, partial indexing
indexing
initialization querying
idle timeworkload knowledge
external tools
human control
Stratos Idreos
database crackingauto-tuning database kernels
incremental, adaptive, partial indexing
indexing
initialization querying
idle timeworkload knowledge
external tools
human control
every query is treated as an advice on how data should be stored
Stratos Idreos
A B C D
...
...
relation1/table1
column-store databasea fixed-width and dense array per attribute
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2
12 7 1
19 3
14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2
12 7 1
19 3
14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2
12 7 1
19 3
14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
sort
1 2 3 4 6 7 8 9
11 12 13 14 16 19
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2
12 7 1
19 3
14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
sort
1 2 3 4 6 7 8 9
11 12 13 14 16 19
binary search
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2
12 7 1
19 3
14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
sort
1 2 3 4 6 7 8 9
11 12 13 14 16 19
binary search re
sult
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2
12 7 1
19 3
14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
sort
1 2 3 4 6 7 8 9
11 12 13 14 16 19
binary search re
sult
time+
knowledge
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
resu
lt
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
resu
lt
gain knowledge on how data is organized
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
dynamically/on-the-fly within the select-operator
resu
lt
gain knowledge on how data is organized
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
Q2: select R.A from R where R.A>7 and R.A<=16
dynamically/on-the-fly within the select-operatorDatabase Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
Q2: select R.A from R where R.A>7 and R.A<=16
dynamically/on-the-fly within the select-operatorDatabase Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
4 2 1 3 6 7 9 8 13 12 11 14 16 19
piece1: A<=7
piece2: 7<A<=10Q2: select R.A from R where R.A>7 and R.A<=16
dynamically/on-the-fly within the select-operatorDatabase Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
4 2 1 3 6 7 9 8 13 12 11 14 16 19
piece1: A<=7
piece2: 7<A<=10
piece3: 10<A<14Q2: select R.A from R where R.A>7 and R.A<=16
dynamically/on-the-fly within the select-operatorDatabase Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
4 2 1 3 6 7 9 8 13 12 11 14 16 19
piece1: A<=7
piece2: 7<A<=10
piece3: 10<A<14
piece4: 14<=A<=16
piece5: A>16
Q2: select R.A from R where R.A>7 and R.A<=16
dynamically/on-the-fly within the select-operatorDatabase Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
4 2 1 3 6 7 9 8 13 12 11 14 16 19
piece1: A<=7
piece2: 7<A<=10
piece3: 10<A<14
piece4: 14<=A<=16
piece5: A>16
Q2: select R.A from R where R.A>7 and R.A<=16
dynamically/on-the-fly within the select-operator
resu
lt
Database Cracking CIDR 2007
Stratos Idreos
13 16 4 9 2 12 7 1 19 3 14 11 8 6
column AQ1: select R.A from R where R.A>10 and R.A<14
4 9 2 7 1 3 8 6 13 12 11 16 19 14
piece1: A<=10
piece2: 10<A<14
piece3: A>=14
4 2 1 3 6 7 9 8 13 12 11 14 16 19
piece1: A<=7
piece2: 7<A<=10
piece3: 10<A<14
piece4: 14<=A<=16
piece5: A>16
Q2: select R.A from R where R.A>7 and R.A<=16
dynamically/on-the-fly within the select-operator
resu
lt
the more we crack, the more we learn
Database Cracking CIDR 2007
Stratos Idreos
select [15,55]
Database Cracking CIDR 2007
Stratos Idreos
select [15,55]
Database Cracking CIDR 2007
Stratos Idreos
10 20 30 40 50 60
select [15,55]
select [15,55]
Database Cracking CIDR 2007
Stratos Idreos
10 20 30 40 50 60
select [15,55]
select [15,55]
Database Cracking CIDR 2007
Stratos Idreos
10 20 30 40 50 60
select [15,55]
select [15,55]
Database Cracking CIDR 2007
Stratos Idreos
10 20 30 40 50 60
select [15,55]
select [15,55]pieces become smaller and smaller
touch at most two pieces at a time
Database Cracking CIDR 2007
Stratos Idreos
# of queries
resp
onse
tim
estandard
optimal
but requires time, workload knowledge & stability
Stratos Idreos
# of queries
resp
onse
tim
estandard
optimal
adaptive
but requires time, workload knowledge & stability
Stratos Idreos
# of queries
resp
onse
tim
estandard
optimal
adaptive
but requires time, workload knowledge & stability
Stratos Idreos
100K random selections random selectivity random value ranges in a 10 million integer column
set-upScan
Full Index
Crack
0.001
0.01
0.1
1
10
100
1000
1 10 100 1000 10000 100000
Resp
onse
tim
e (
secs
)
Query sequence (x1000)
continuous adaptation
Database Cracking CIDR 2007
Stratos Idreos
100K random selections random selectivity random value ranges in a 10 million integer column
almost no initialization overhead
set-upScan
Full Index
Crack
0.001
0.01
0.1
1
10
100
1000
1 10 100 1000 10000 100000
Resp
onse
tim
e (
secs
)
Query sequence (x1000)
continuous adaptation
Database Cracking CIDR 2007
Stratos Idreos
100K random selections random selectivity random value ranges in a 10 million integer column
almost no initialization overhead
continuous improvement
set-upScan
Full Index
Crack
0.001
0.01
0.1
1
10
100
1000
1 10 100 1000 10000 100000
Resp
onse
tim
e (
secs
)
Query sequence (x1000)
continuous adaptation
Database Cracking CIDR 2007
Stratos Idreos
100K random selections random selectivity random value ranges in a 10 million integer column
almost no initialization overhead
continuous improvement
set-upScan
Full Index
Crack
0.001
0.01
0.1
1
10
100
1000
1 10 100 1000 10000 100000
Resp
onse
tim
e (
secs
)
Query sequence (x1000)
continuous adaptation
Database Cracking CIDR 2007
Stratos Idreos
10K random selections selectivity 10% random value ranges in a 30 million integer column
Database Cracking CIDR 2007
set-up
0.004
200
0.001
0.01
0.1
1
10
100
1 10 100 1000 10000
Cum
ula
tive a
vera
ge r
esp
onse
tim
e (
secs
)
Query sequence
Scan
Full Index
Crack
continuous adaptation
Stratos Idreos
10K random selections selectivity 10% random value ranges in a 30 million integer column
10K queries later, Full Index still has not
amortized the initialization costs
Database Cracking CIDR 2007
set-up
0.004
200
0.001
0.01
0.1
1
10
100
1 10 100 1000 10000
Cum
ula
tive a
vera
ge r
esp
onse
tim
e (
secs
)
Query sequence
Scan
Full Index
Crack
continuous adaptation
Stratos Idreos
Atable1
Stratos Idreos
A B C D
...
...
table1
Stratos Idreos
A B C D
...
...
table1
select R.A from R where R.A>10 and R.A<14
Stratos Idreos
A B C D
...
...
table1
select R.A from R where R.A>10 and R.A<14select max(R.A),max(R.B),max(S.A),max(S.B) from R,S where v1 <R.C<v2 and v3 <R.D<v4 and v5 <R.E<v6 and k1 <S.C<k2 and k3 <S.D<k4 and k5 <S.E<k6 and R.F = S.F
Stratos Idreos
A B C D
...
...
table1
select R.A from R where R.A>10 and R.A<14select max(R.A),max(R.B),max(S.A),max(S.B) from R,S where v1 <R.C<v2 and v3 <R.D<v4 and v5 <R.E<v6 and k1 <S.C<k2 and k3 <S.D<k4 and k5 <S.E<k6 and R.F = S.F
updatesjoins
concurrency control
...
cracking databases
updates (SIGMOD07)
storage-restrictions (SIGMOD09)
benchmarking (TPCTC10)
concurrency control
(PVLDB12)
algorithms (PVLDB11)
basics (CIDR07)
multi-cores (SIGMOD15)
hadoop (Yale/Saarland)
b-trees (HP Labs)
>1 columns (SIGMOD09)
robustness (PVLDB12)
adaptive storage
(SIGMOD14)
time-series (SIGMOD14)
encryption (SIGMOD16)
Stratos Idreos
updates
Updates SIGMOD 2007
Stratos Idreos
cracking indices are auxiliary data structurescan be dropped any time
Stratos Idreos
column of 10M tuples, random queries 1000 random insertions every 1000 queries
crack (forget) scan
forgettingwhen updates arrive, drop the index
Stratos Idreos
column of 10M tuples, random queries 1000 random insertions every 1000 queries
queries (x1000) queries (x1000)
crack (forget) scan
we do not exploit past cracking
forgettingwhen updates arrive, drop the index
Stratos Idreos
column of 10M tuples, random queries 1000 random insertions every 1000 queries
crack (forget) scan
we do not exploit past cracking
forgettingwhen updates arrive, drop the index
Stratos Idreos
column of 10M tuples, random queries 1000 random insertions every 1000 queries
queries (x1000) queries (x1000)
crack (forget) scan
we do not exploit past cracking
forgettingwhen updates arrive, drop the index
Stratos Idreos
log updates and apply on-line and on-demand during cracking
cracking updates
maintain
updates query
Stratos Idreos
log updates and apply on-line and on-demand during cracking
cracking updates
maintain
updates query
Stratos Idreos
pieces are ordered
values in a piece are not ordered
goal: minimize physical actions
arrays are dense3
column
pending inserts
pending deletes
1
9
17
12
16
25
22
29
33
39
31
10
20
30
Stratos Idreos
pieces are ordered
values in a piece are not ordered
goal: minimize physical actions
arrays are dense3
column
pending inserts
pending deletes
1
9
17
12
16
25
22
29
33
39
31
10
20
30
11
Stratos Idreos
pieces are ordered
values in a piece are not ordered
goal: minimize physical actions
arrays are dense3
column
pending inserts
pending deletes
1
9
17
12
16
25
22
29
33
39
31
10
20
30
11
Stratos Idreos
pieces are ordered
values in a piece are not ordered
goal: minimize physical actions
arrays are dense3
column
pending inserts
pending deletes
1
9
17
12
16
25
22
29
33
39
31
10
20
30
11
Stratos Idreos
pieces are ordered
values in a piece are not ordered
goal: minimize physical actions
arrays are dense3
column
pending inserts
pending deletes
1
9
17
12
16
25
22
29
33
39
31
10
20
30
11
Stratos Idreos
pieces are ordered
values in a piece are not ordered
goal: minimize physical actions
arrays are dense3
column
pending inserts
pending deletes
1
9
17
12
16
25
22
29
33
39
31
10
20
30
11
Stratos Idreos
pieces are ordered
values in a piece are not ordered
goal: minimize physical actions
arrays are dense3
column
pending inserts
pending deletes
1
9
17
12
16
25
22
29
33
39
31
10
20
30
11
Stratos Idreos
pieces are ordered
values in a piece are not ordered
goal: minimize physical actions
arrays are dense3
column
pending inserts
pending deletes
1
9
17
12
16
25
22
29
33
39
31
10
20
30
11
Stratos Idreos
column of 10M tuples, random queries 1000 random insertions every 1000 queries
merge all updates merge only qualifying updates
Stratos Idreos
column of 10M tuples, random queries 1000 random insertions every 1000 queries
queries (x1000) queries (x1000)
merge all updates merge only qualifying updates
gradual merging avoids high picks
Stratos Idreos
column of 10M tuples, random queries 1000 random insertions every 1000 queries
merge all updates merge only qualifying updates
gradual merging avoids high picks
Stratos Idreos
column of 10M tuples, random queries 1000 random insertions every 1000 queries
queries (x1000) queries (x1000)
merge all updates merge only qualifying updates
gradual merging avoids high picks
Stratos Idreos
the ripplecolumn
pending updates
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
column
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
column
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
column
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
column
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
column
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
column
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
column
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
column
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
column
quer
y re
sult
Stratos Idreos
the ripplecolumn
pending updates
quer
y re
sult
doing work not relevant for the current query
column
quer
y re
sult
update only the hot tuples
Stratos Idreos
forget ripple
Stratos Idreos
queries (x1000) queries (x1000)
merge all updates merge only qualifying updates
forget ripple
maintains adaptive behavior
Stratos Idreos
sideways crackinghow to process >1 columns
Sideways Cracking, SIGMOD 09
Stratos Idreos
column-stores
a1a2a3a4a5a6
b1b2b3b4b5b6
c1c2c3c4c5c6
positional alignment
positional lookupsA(i) = A + i * width(A)
Tuple 1Tuple 2Tuple 3Tuple 4Tuple 5Tuple 6
A B C
Sideways Cracking, SIGMOD 09
Stratos Idreos
select min(C) from R where A<10 & B<20
B<20 minCA<10A B C D IDs B CIDsdisk memory
Sideways Cracking, SIGMOD 09
Stratos Idreos
pos1 pos2
1 0 1 0
A
sort
edselect max(D),min(E) from R where (A>10 and A<40) and (B>20 and B<60)
binary search for 10 & 40
B
for all B values between pos1 & 2: if B>20 and B<60 mark bit vector at pos i
maxD
D
for each marked position max(D)
…
avoid scan of A avoid TR on B work on a restricted area across all columns good for memory hierarchy
Stratos Idreos
a1 a2 a3 a4 a5 a6
1 3 6
RowIDs
a1 a2 a3 a4 a5 a6
6 1 3
RowIDs
ordered
unordered
sequential access
random access
without cracking
with cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A
sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A
sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C D
sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C Dquery sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C Dquery sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C Dquery sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C Dquery sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C Dquery sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C D
query
sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C D
query
sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C D
query
sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C D
query
sideways cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
A B C D
log crack actions and replay to align columns dynamically
sideways cracking
replace tuple reconstruction with cracking actions Sideways Cracking, SIGMOD 09
Stratos Idreos
sideways cracking
12
3
5
9
15
22
7
26
4
2
24
11
16
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
A B
select B from R where 10<A<15
4
3
5
9
2
7
12
11
15
22
24
26
16
b9
b2
b3
b4
b10
b7
b1
b12
b5
b6
b11
b8
b13
AB
Position 1
value <=10Piece 1
Position 7
value >10Piece 2
Position 9
value >=15Piece 3
select B from R where 5=<A<17
4
3
2
9
5
7
12
11
15
16
24
26
22
b9
b2
b10
b4
b3
b7
b1
b12
b5
b13
b11
b8
b6
Position 1
value <5Piece 1
Position 7
value >10Piece 3
Position 9
value >=15Piece 4
Position 4
value >=5Piece 2
Position 11
value >=17Piece 5
Cracker index Cracker indexM ABMInitial state
Sideways Cracking, SIGMOD 09
Stratos Idreos
sideways cracking
12
3
5
9
15
22
7
26
4
2
24
11
16
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
A B
select B from R where 10<A<15
4
3
5
9
2
7
12
11
15
22
24
26
16
b9
b2
b3
b4
b10
b7
b1
b12
b5
b6
b11
b8
b13
AB
Position 1
value <=10Piece 1
Position 7
value >10Piece 2
Position 9
value >=15Piece 3
select B from R where 5=<A<17
4
3
2
9
5
7
12
11
15
16
24
26
22
b9
b2
b10
b4
b3
b7
b1
b12
b5
b13
b11
b8
b6
Position 1
value <5Piece 1
Position 7
value >10Piece 3
Position 9
value >=15Piece 4
Position 4
value >=5Piece 2
Position 11
value >=17Piece 5
Cracker index Cracker indexM ABMInitial state
Sideways Cracking, SIGMOD 09
Stratos Idreos
sideways cracking
12
3
5
9
15
22
7
26
4
2
24
11
16
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
A B
select B from R where 10<A<15
4
3
5
9
2
7
12
11
15
22
24
26
16
b9
b2
b3
b4
b10
b7
b1
b12
b5
b6
b11
b8
b13
AB
Position 1
value <=10Piece 1
Position 7
value >10Piece 2
Position 9
value >=15Piece 3
select B from R where 5=<A<17
4
3
2
9
5
7
12
11
15
16
24
26
22
b9
b2
b10
b4
b3
b7
b1
b12
b5
b13
b11
b8
b6
Position 1
value <5Piece 1
Position 7
value >10Piece 3
Position 9
value >=15Piece 4
Position 4
value >=5Piece 2
Position 11
value >=17Piece 5
Cracker index Cracker indexM ABMInitial state
Sideways Cracking, SIGMOD 09
Stratos Idreos
sideways cracking
12
3
5
9
15
22
7
26
4
2
24
11
16
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
A B
select B from R where 10<A<15
4
3
5
9
2
7
12
11
15
22
24
26
16
b9
b2
b3
b4
b10
b7
b1
b12
b5
b6
b11
b8
b13
AB
Position 1
value <=10Piece 1
Position 7
value >10Piece 2
Position 9
value >=15Piece 3
select B from R where 5=<A<17
4
3
2
9
5
7
12
11
15
16
24
26
22
b9
b2
b10
b4
b3
b7
b1
b12
b5
b13
b11
b8
b6
Position 1
value <5Piece 1
Position 7
value >10Piece 3
Position 9
value >=15Piece 4
Position 4
value >=5Piece 2
Position 11
value >=17Piece 5
Cracker index Cracker indexM ABMInitial state
Cracker MapHead Tail
Sideways Cracking, SIGMOD 09
Stratos Idreos
sideways cracking
12
3
5
9
15
22
7
26
4
2
24
11
16
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
A B
select B from R where 10<A<15
4
3
5
9
2
7
12
11
15
22
24
26
16
b9
b2
b3
b4
b10
b7
b1
b12
b5
b6
b11
b8
b13
AB
Position 1
value <=10Piece 1
Position 7
value >10Piece 2
Position 9
value >=15Piece 3
select B from R where 5=<A<17
4
3
2
9
5
7
12
11
15
16
24
26
22
b9
b2
b10
b4
b3
b7
b1
b12
b5
b13
b11
b8
b6
Position 1
value <5Piece 1
Position 7
value >10Piece 3
Position 9
value >=15Piece 4
Position 4
value >=5Piece 2
Position 11
value >=17Piece 5
Cracker index Cracker indexM ABMInitial state
Cracker MapHead Tail
crack based on head, carry tail
Sideways Cracking, SIGMOD 09
Stratos Idreos
sideways cracking
12
3
5
9
15
22
7
26
4
2
24
11
16
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
A B
select B from R where 10<A<15
4
3
5
9
2
7
12
11
15
22
24
26
16
b9
b2
b3
b4
b10
b7
b1
b12
b5
b6
b11
b8
b13
AB
Position 1
value <=10Piece 1
Position 7
value >10Piece 2
Position 9
value >=15Piece 3
select B from R where 5=<A<17
4
3
2
9
5
7
12
11
15
16
24
26
22
b9
b2
b10
b4
b3
b7
b1
b12
b5
b13
b11
b8
b6
Position 1
value <5Piece 1
Position 7
value >10Piece 3
Position 9
value >=15Piece 4
Position 4
value >=5Piece 2
Position 11
value >=17Piece 5
Cracker index Cracker indexM ABMInitial state
Cracking knowledge
Cracker MapHead Tail
crack based on head, carry tail
Sideways Cracking, SIGMOD 09
Stratos Idreos
sideways cracking
12
3
5
9
15
22
7
26
4
2
24
11
16
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
A B
select B from R where 10<A<15
4
3
5
9
2
7
12
11
15
22
24
26
16
b9
b2
b3
b4
b10
b7
b1
b12
b5
b6
b11
b8
b13
AB
Position 1
value <=10Piece 1
Position 7
value >10Piece 2
Position 9
value >=15Piece 3
select B from R where 5=<A<17
4
3
2
9
5
7
12
11
15
16
24
26
22
b9
b2
b10
b4
b3
b7
b1
b12
b5
b13
b11
b8
b6
Position 1
value <5Piece 1
Position 7
value >10Piece 3
Position 9
value >=15Piece 4
Position 4
value >=5Piece 2
Position 11
value >=17Piece 5
Cracker index Cracker indexM ABMInitial state
Cracking knowledge No tuple reconstruction
Cracker MapHead Tail
crack based on head, carry tail
Sideways Cracking, SIGMOD 09
Stratos Idreos
sideways cracking
12
3
5
9
15
22
7
26
4
2
24
11
16
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
A B
select B from R where 10<A<15
4
3
5
9
2
7
12
11
15
22
24
26
16
b9
b2
b3
b4
b10
b7
b1
b12
b5
b6
b11
b8
b13
AB
Position 1
value <=10Piece 1
Position 7
value >10Piece 2
Position 9
value >=15Piece 3
select B from R where 5=<A<17
4
3
2
9
5
7
12
11
15
16
24
26
22
b9
b2
b10
b4
b3
b7
b1
b12
b5
b13
b11
b8
b6
Position 1
value <5Piece 1
Position 7
value >10Piece 3
Position 9
value >=15Piece 4
Position 4
value >=5Piece 2
Position 11
value >=17Piece 5
Cracker index Cracker indexM ABMInitial state
Dynamically/on-the-fly within the select-operatorCracking knowledge No tuple reconstruction
Cracker MapHead Tail
crack based on head, carry tail
Sideways Cracking, SIGMOD 09
Stratos Idreos
sideways cracking
12
3
5
9
15
22
7
26
4
2
24
11
16
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
A B
select B from R where 10<A<15
4
3
5
9
2
7
12
11
15
22
24
26
16
b9
b2
b3
b4
b10
b7
b1
b12
b5
b6
b11
b8
b13
AB
Position 1
value <=10Piece 1
Position 7
value >10Piece 2
Position 9
value >=15Piece 3
select B from R where 5=<A<17
4
3
2
9
5
7
12
11
15
16
24
26
22
b9
b2
b10
b4
b3
b7
b1
b12
b5
b13
b11
b8
b6
Position 1
value <5Piece 1
Position 7
value >10Piece 3
Position 9
value >=15Piece 4
Position 4
value >=5Piece 2
Position 11
value >=17Piece 5
Cracker index Cracker indexM ABMInitial state
Dynamically/on-the-fly within the select-operatorCracking knowledge No tuple reconstruction
Cracker MapHead Tail
crack based on head, carry tail
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
perform the same cracks and in the same order on all maps with the same head
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
perform the same cracks and in the same order on all maps with the same head
on-line alignment touch/load everything, always
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
perform the same cracks and in the same order on all maps with the same head
on-line alignment Xtouch/load everything, always
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
perform the same cracks and in the same order on all maps with the same head
remember and replay cracks across columns
on-line alignment Xtouch/load everything, always
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
Align
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
Align
Align
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
Align
AlignCorrect
alignment
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
7
4
1
2
8
3
6
B
b1
b2
b3
b4
b5
b6
b7
A
c1
c2
c3
c4
c5
c6
c7
C
select B from R where A<3 select C from R where A<5 select B,C from R where A<4
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
3
4
1
2
8
7
6
c6
c2
c3
c4
c5
c1
c7
v<5
v>=5
2
1
3
7
8
4
6
M
b4
b3
b6
b1
b5
b2
b7
v<3
v>=4
v>=3
3
2
1
4
8
7
6
c6
c4
c3
c2
c5
c1
c7
v<4
v>=5
v>=4
Result
b4
b3
Result
c6
c2
c3
c4
Result
b4
b3
b6
c6
c4
c3
Initial state
Without Alignment
2
1
4
7
8
3
6
b4
b3
b2
b1
b5
b6
b7
v<3
v>=3
b4
b3
Result
WithAlignment
2
1
4
3
8
7
6
c4
c3
c2
c6
c5
c1
c7
v<3
v>=5
Result
c4
c3
c2
c6
2
1
4
7
8
3
6
c4
c3
c2
c1
c5
c6
c7
v<3
v>=3
Crack A<5
v>=3
2
1
4
3
8
7
6
b4
b3
b2
b6
b5
b1
b7
v<3
v>=5
v>=3
2
1
3
4
8
7
6
b4
b3
b6
b2
b5
b1
b7
v<3
v>=5
v>=3
v>=4
2
1
3
4
8
7
6
c4
c3
c6
c2
c5
c1
c7
v<3
v>=5
v>=3
v>=4
Result
b4
b3
b6
c4
c3
c6
Wrong
allignment
Correct
allignment
Crack A<3
Crack A<5Crack A<3
Crack A<3 Crack A<4Crack A<5
Crack A<4
AB
MAC
MAB
MAC
MAC MAC MAC
MABMABMAB
Wrong alignment
Align
AlignCorrect
alignment
replace tuple reconstruction with cracking
adaptive alignment
Sideways Cracking, SIGMOD 09
Stratos Idreos
multi-selections
select D from R where 3<A<10 and 4<B<8 and 1<C<7
wider maps...but too many combinations maps of different maps sets lead to alignment problems
Sideways Cracking, SIGMOD 09
Stratos Idreos
multi-selections
use a single map set and exploit bit-vectors
select D from R where 3<A<10 and 4<B<8 and 1<C<7
wider maps...but too many combinations maps of different maps sets lead to alignment problems
Sideways Cracking, SIGMOD 09
Stratos Idreos
multi-selections
12
3
5
9
8
22
7
26
4
2
7
9
2
6
10
7
11
16
2
5
8
3
A B
select_create_bv(A,3,10,B,4,8)
2
3
5
9
8
7
7
4
26
12
22
8
2
6
10
7
3
16
5
2
9
11
MAB
v<=3
v>3
v>=10
1
0
1
0
0
1
Bit
vector bv
3
6
2
1
6
9
12
2
11
17
3
C
select_refine_bv(A,3,10,C,1,7,bv)
2
3
5
9
8
7
7
4
26
12
22
17
6
2
1
6
3
12
11
2
3
9
v<=3
v>3
v>=10
1
0
1
0
0
0
Bit
vector bv
9
4
2
10
12
19
3
6
5
8
1
D
reconstruct(A,3,10,D,bv)
2
3
5
9
8
7
7
4
26
12
22
8
4
2
10
12
1
3
5
6
9
19
v<=3
v>3
v>=10
2
12
Result
MAC MAD
select D from R
where 3<A<10 and
4<B<8 and 1<C<7
Initial stateQuery
1
0
1
0
0
0
Bit
vector bv
select D from R where 3<A<10 and 4<B<8 and 1<C<7
Sideways Cracking, SIGMOD 09
Stratos Idreos
multi-selections
12
3
5
9
8
22
7
26
4
2
7
9
2
6
10
7
11
16
2
5
8
3
A B
select_create_bv(A,3,10,B,4,8)
2
3
5
9
8
7
7
4
26
12
22
8
2
6
10
7
3
16
5
2
9
11
MAB
v<=3
v>3
v>=10
1
0
1
0
0
1
Bit
vector bv
3
6
2
1
6
9
12
2
11
17
3
C
select_refine_bv(A,3,10,C,1,7,bv)
2
3
5
9
8
7
7
4
26
12
22
17
6
2
1
6
3
12
11
2
3
9
v<=3
v>3
v>=10
1
0
1
0
0
0
Bit
vector bv
9
4
2
10
12
19
3
6
5
8
1
D
reconstruct(A,3,10,D,bv)
2
3
5
9
8
7
7
4
26
12
22
8
4
2
10
12
1
3
5
6
9
19
v<=3
v>3
v>=10
2
12
Result
MAC MAD
select D from R
where 3<A<10 and
4<B<8 and 1<C<7
Initial stateQuery
1
0
1
0
0
0
Bit
vector bv
select D from R where 3<A<10 and 4<B<8 and 1<C<7
Crack 3<A<10 Analyze tail 4<B<8 Create bit vector
Sideways Cracking, SIGMOD 09
Stratos Idreos
multi-selections
12
3
5
9
8
22
7
26
4
2
7
9
2
6
10
7
11
16
2
5
8
3
A B
select_create_bv(A,3,10,B,4,8)
2
3
5
9
8
7
7
4
26
12
22
8
2
6
10
7
3
16
5
2
9
11
MAB
v<=3
v>3
v>=10
1
0
1
0
0
1
Bit
vector bv
3
6
2
1
6
9
12
2
11
17
3
C
select_refine_bv(A,3,10,C,1,7,bv)
2
3
5
9
8
7
7
4
26
12
22
17
6
2
1
6
3
12
11
2
3
9
v<=3
v>3
v>=10
1
0
1
0
0
0
Bit
vector bv
9
4
2
10
12
19
3
6
5
8
1
D
reconstruct(A,3,10,D,bv)
2
3
5
9
8
7
7
4
26
12
22
8
4
2
10
12
1
3
5
6
9
19
v<=3
v>3
v>=10
2
12
Result
MAC MAD
select D from R
where 3<A<10 and
4<B<8 and 1<C<7
Initial stateQuery
1
0
1
0
0
0
Bit
vector bv
select D from R where 3<A<10 and 4<B<8 and 1<C<7
Crack 3<A<10 Analyze tail 4<B<8 Create bit vector
Align 3<A<10 Analyze tail 1<C<7 Refine bit vector
Sideways Cracking, SIGMOD 09
Stratos Idreos
multi-selections
12
3
5
9
8
22
7
26
4
2
7
9
2
6
10
7
11
16
2
5
8
3
A B
select_create_bv(A,3,10,B,4,8)
2
3
5
9
8
7
7
4
26
12
22
8
2
6
10
7
3
16
5
2
9
11
MAB
v<=3
v>3
v>=10
1
0
1
0
0
1
Bit
vector bv
3
6
2
1
6
9
12
2
11
17
3
C
select_refine_bv(A,3,10,C,1,7,bv)
2
3
5
9
8
7
7
4
26
12
22
17
6
2
1
6
3
12
11
2
3
9
v<=3
v>3
v>=10
1
0
1
0
0
0
Bit
vector bv
9
4
2
10
12
19
3
6
5
8
1
D
reconstruct(A,3,10,D,bv)
2
3
5
9
8
7
7
4
26
12
22
8
4
2
10
12
1
3
5
6
9
19
v<=3
v>3
v>=10
2
12
Result
MAC MAD
select D from R
where 3<A<10 and
4<B<8 and 1<C<7
Initial stateQuery
1
0
1
0
0
0
Bit
vector bv
select D from R where 3<A<10 and 4<B<8 and 1<C<7
Crack 3<A<10 Analyze tail 4<B<8 Create bit vector
Align 3<A<10 Analyze tail 1<C<7 Refine bit vector
Align 3<A<10 Grab tail values
Sideways Cracking, SIGMOD 09
Stratos Idreos
multi-selections
12
3
5
9
8
22
7
26
4
2
7
9
2
6
10
7
11
16
2
5
8
3
A B
select_create_bv(A,3,10,B,4,8)
2
3
5
9
8
7
7
4
26
12
22
8
2
6
10
7
3
16
5
2
9
11
MAB
v<=3
v>3
v>=10
1
0
1
0
0
1
Bit
vector bv
3
6
2
1
6
9
12
2
11
17
3
C
select_refine_bv(A,3,10,C,1,7,bv)
2
3
5
9
8
7
7
4
26
12
22
17
6
2
1
6
3
12
11
2
3
9
v<=3
v>3
v>=10
1
0
1
0
0
0
Bit
vector bv
9
4
2
10
12
19
3
6
5
8
1
D
reconstruct(A,3,10,D,bv)
2
3
5
9
8
7
7
4
26
12
22
8
4
2
10
12
1
3
5
6
9
19
v<=3
v>3
v>=10
2
12
Result
MAC MAD
select D from R
where 3<A<10 and
4<B<8 and 1<C<7
Initial stateQuery
1
0
1
0
0
0
Bit
vector bv
select D from R where 3<A<10 and 4<B<8 and 1<C<7
Crack 3<A<10 Analyze tail 4<B<8 Create bit vector
Align 3<A<10 Analyze tail 1<C<7 Refine bit vector
Align 3<A<10 Grab tail values
Use histogram-like info from maps to choose map set
Sideways Cracking, SIGMOD 09
Stratos Idreos
70
330
100
150
200
250
300
0 5 10 15 20 25 30Query sequence
TPC-H Query 15764420
1000
10000
Sel. Crack
Sid. Crack
MonetDB
PresortedMySQL
Presorted
Resp
onse
tim
e (m
illi s
ecs)
Sideways Cracking, SIGMOD 09
Stratos Idreos
70
330
100
150
200
250
300
0 5 10 15 20 25 30Query sequence
TPC-H Query 15764420
1000
10000
Sel. Crack
Sid. Crack
MonetDB
PresortedMySQL
Presorted
Resp
onse
tim
e (m
illi s
ecs)
normal MonetDB
selection cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
70
330
100
150
200
250
300
0 5 10 15 20 25 30Query sequence
TPC-H Query 15764420
1000
10000
Sel. Crack
Sid. Crack
MonetDB
PresortedMySQL
Presorted
preparation cost 3-14 minutes
per query
Resp
onse
tim
e (m
illi s
ecs)
presorted MonetDB
normal MonetDB
selection cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
70
330
100
150
200
250
300
0 5 10 15 20 25 30Query sequence
TPC-H Query 15764420
1000
10000
Sel. Crack
Sid. Crack
MonetDB
PresortedMySQL
Presorted
preparation cost 3-14 minutes
per query
Resp
onse
tim
e (m
illi s
ecs)
presorted MonetDBMonetDB with
sideways cracking
normal MonetDB
selection cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
70
330
100
150
200
250
300
0 5 10 15 20 25 30Query sequence
TPC-H Query 15764420
1000
10000
Sel. Crack
Sid. Crack
MonetDB
PresortedMySQL
Presorted
preparation cost 3-14 minutes
per query
Resp
onse
tim
e (m
illi s
ecs)
presorted MonetDBMonetDB with
sideways cracking
normal MonetDB
selection cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
70
330
100
150
200
250
300
0 5 10 15 20 25 30Query sequence
TPC-H Query 15764420
1000
10000
Sel. Crack
Sid. Crack
MonetDB
PresortedMySQL
Presorted
preparation cost 3-14 minutes
per query
Resp
onse
tim
e (m
illi s
ecs)
presorted MonetDBMonetDB with
sideways cracking
normal MonetDB
selection cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
70
330
100
150
200
250
300
0 5 10 15 20 25 30Query sequence
TPC-H Query 15764420
1000
10000
Sel. Crack
Sid. Crack
MonetDB
PresortedMySQL
Presorted
preparation cost 3-14 minutes
per query
Resp
onse
tim
e (m
illi s
ecs)
presorted MonetDBMonetDB with
sideways cracking
normal MonetDB
selection cracking
Sideways Cracking, SIGMOD 09
Stratos Idreos
Partial Sideways Cracking
15
8
19
6
11
2
14
5
12
18
4
9
13
7
b1
b2
b3
b4
b5
b6
b7
b8
b9
b10
b11
b12
b13
b14
A B
Select B from R where 9<A<=15Initial state
Chunk map H (A,id)
v<=9
U
v>9
F
v>15
U
v<=9
E
v>9
M,C=1
v>15
E
Partial map M
Select B from R where 9<A<13
v<=9
E
v>9
M,C=1
v>15
E
v>=13
M,C=1
Select B from R where 5<=A<8
v<5
U
v>9
F
v>15
Uv>15, E
v>9
M,C=1
v>=13
M,C=1
v>=5
F
v>=8
U
v<5, E
v>=5
M,C=2
v>=8, E
7
8
9
6
4
2
5
14
12
13
15
11
19
18
14
2
12
4
11
6
8
7
9
13
1
5
3
10
14
12
13
15
11
b7
b9
b13
b1
b5
11
12
13
15
14
b5
b9
b13
b1
b7
4
2
7
6
5
9
8
14
12
13
15
11
19
18
11
6
14
11
8
12
2
7
9
13
1
5
3
10
11
12
13
15
14
b5
b9
b13
b1
b7
7
6
5
b14
b11
b8
c1
c2
c3
c4
c5
c6
c7
c8
c9
c10
c11
c12
c13
c14
C
Select C from R where 8<=A<15
v<5
U
v>9
F
v>15
U v>15, E
v>9
M,C=1
v>=13
M,C=1
v>=5
F
v>=8
F
v<8, E
v>=8
M,C=3
4
2
7
6
5
9
8
14
12
13
15
11
19
18
11
6
14
11
8
12
2
7
9
13
1
5
3
10
11
12
13
14
15
c5
c9
c13
c7
c1
9
8
c12
c2
v>15
M,C=1
A
Chunk map H (A,id)A Chunk map H (A,id)A
AB Partial map MAB
Partial map MAB Partial map MAC
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Keys
Stratos Idreos
Stratos Idreos
crack
crack
Stratos Idreos
crack
crack
Stratos Idreos
crack
crack
no alignment
no alignment
Stratos Idreos
crack
crack
full alignment
full alignment
no alignment
no alignment
Stratos Idreos
crack
crack
full alignment
full alignment
partial alignment
partial alignment
no alignment
no alignment
Stratos Idreos
concurrency control
Concurrency Control, PVLDB 12
problem: read queries become write queries! (?)
goal: be able to crack for multiple queries in parallel
Stratos Idreos
traditional indexing adaptive indexing
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexingwrite queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
change index contents and structure
only index structure changes
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
change index contents and structure
only index structure changes
no need for traditional locks = too heavy
short term latches = fast and release quickly
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
all or nothing incremental and optional
change index contents and structure
only index structure changes
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
all or nothing incremental and optional
change index contents and structure
only index structure changes
v1 v2 v3
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
all or nothing incremental and optional
change index contents and structure
only index structure changes
v1 v2 v3
a b
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
all or nothing incremental and optional
change index contents and structure
only index structure changes
v1 v2 v3 v1 v2 v3
a b
a b
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
all or nothing incremental and optional
change index contents and structure
only index structure changes
v1 v2 v3 v1 v2 v3
a b
a b
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
all or nothing incremental and optional
change index contents and structure
only index structure changes
v1 v2 v3 v1 v2 v3
a b
a b
v1 v2 v3a
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
all or nothing incremental and optional
change index contents and structure
only index structure changes
v1 v2 v3 v1 v2 v3
a b
a b
v1 v2 v3a
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
all or nothing incremental and optional
impact stable storage stable storage optional
change index contents and structure
only index structure changes
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
all or nothing incremental and optional
impact stable storage stable storage optional
change index contents and structure
only index structure changes
v1 v2 v3 v1 v2 v3a b
write queries read queries
a b
Concurrency Control, PVLDB 12
Stratos Idreos
traditional indexing adaptive indexing
all or nothing incremental and optional
impact stable storage stable storage optional
change index contents and structure
only index structure changes
need to serialize can execute in any order
write queries read queries
Concurrency Control, PVLDB 12
Stratos Idreos
fewer conflicts as we adapt
query sequence
confl
icts
Scan
Full Index
Crack
0.001
0.01
0.1
1
10
100
1000
1 10 100 1000 10000 100000
Resp
onse
tim
e (
secs
)
Query sequence (x1000)
Concurrency Control, PVLDB 12
Stratos Idreos
late reconstruction
select min(C) from R where A<10 & B<20
diskA B C D
Concurrency Control, PVLDB 12
Stratos Idreos
late reconstruction
select min(C) from R where A<10 & B<20
A<10disk memory
A B C D
Concurrency Control, PVLDB 12
Stratos Idreos
late reconstruction
select min(C) from R where A<10 & B<20
A<10disk memory
A B C D IDs
Concurrency Control, PVLDB 12
Stratos Idreos
late reconstruction
select min(C) from R where A<10 & B<20
A<10disk memory
A B C D IDs B
Concurrency Control, PVLDB 12
Stratos Idreos
late reconstruction
select min(C) from R where A<10 & B<20
B<20A<10disk memory
A B C D IDs B
Concurrency Control, PVLDB 12
Stratos Idreos
late reconstruction
select min(C) from R where A<10 & B<20
B<20A<10disk memory
A B C D IDs B CIDs
Concurrency Control, PVLDB 12
Stratos Idreos
late reconstruction
select min(C) from R where A<10 & B<20
B<20 minCA<10disk memory
A B C D IDs B CIDs
Concurrency Control, PVLDB 12
Stratos Idreos
late reconstruction
select min(C) from R where A<10 & B<20
B<20 minCA<10disk memory
A B C D IDs B CIDs
column lock and release as soon as an operator completes
Concurrency Control, PVLDB 12
Stratos Idreos Concurrency Control, PVLDB 12
select [a,b]
Stratos Idreos
need to latch only to be cracked pieces (max 2 per select)
Concurrency Control, PVLDB 12
select [a,b]
Stratos Idreos
piece locking
avl-tree crack column
Concurrency Control, PVLDB 12
Stratos Idreos
piece locking
avl-tree crack columncrack select
Concurrency Control, PVLDB 12
Stratos Idreos
piece locking
avl-tree crack columncrack selectwlock
Concurrency Control, PVLDB 12
Stratos Idreos
piece locking
avl-tree crack columncrack select
wlock
Concurrency Control, PVLDB 12
Stratos Idreos
piece locking
avl-tree crack column
Concurrency Control, PVLDB 12
Stratos Idreos
piece locking
avl-tree crack columnmax
Concurrency Control, PVLDB 12
Stratos Idreos
piece locking
avl-tree crack columnmaxrlock
Concurrency Control, PVLDB 12
Stratos Idreos
piece locking
avl-tree crack columnmax
rlock
Concurrency Control, PVLDB 12
Stratos Idreos
piece locking
avl-tree crack columnmax
rlock
Concurrency Control, PVLDB 12
Stratos Idreos
piece locking
avl-tree crack columnmax
rlock
Concurrency Control, PVLDB 12
Stratos Idreos
avl-tree crack column1090
140
200
300
piece locking
Stratos Idreos
avl-tree crack column crack select1090
140
200
300
65
230
piece locking
Stratos Idreos
avl-tree crack column crack selectwlock
wlock
1090
140
200
300
65
230
piece locking
Stratos Idreos
avl-tree crack column crack selectwlock
wlock
r/wlock
1090
140
200
300
65
230
piece locking
Stratos Idreos
avl-tree crack column crack selectwlock
wlock
q1,q2,q3...qn1090
140
200
300
65
230
10
90
piece locking
Stratos Idreos
avl-tree crack column crack selectwlock
wlock
q1,q2,q3...qn1090
140
200
300
65
230
10
90
2030
7080
piece locking
Stratos Idreos
avl-tree crack column crack selectwlock
wlock
q1,q2,q3...qn1090
140
200
300
65
230
10
90
2030
7080
wlock
piece locking
Stratos Idreos
0
10
20
30
40
50
60
70
80
1 2 4 8 16 32
Tota
l Tim
e for
1024 Q
ueries
(secs
)
Number of Concurrent Clients
scansort
crack
300
0
50
100
150
200
250
350
1 2 4 8 16 32
Thro
ughput ove
r 1024 Q
ueries
(Queries
/ se
c)
Number of Concurrent Clients
scansort
crack
4.5
4.55
4.6
4.65
4.7
4.75
4.8
Tota
l Tim
e for
1024 Q
ueries
(secs
)
(Sequential Execution)
Concurrency Control
Enabled Disabled
Rows: 100M Query: select sum(A) from R where v1 < A1 < v2 Selectivity: 0.01%, Random, # of queries: 1024 Clients: 1-32, Machine: 4 cores
adaptive indexing maintains its performance advantage
Concurrency Control, PVLDB 12
Stratos Idreos
crack time and wait time
10-6
10-5
10-4
10-3
10-2
10-1
100
1 10 100 1000
Tim
e p
er
query
(se
cs)
Query sequence
index refinementwait time
Query: select sum(A) from R where v1 < A1 < v2 Selectivity: 0.01%, Random, # of queries: 1024 Clients: 8, Machine: 4 cores
adaptive indexing maintains
the adaptive behavior
adaptive behavior also for conflicts
Concurrency Control, PVLDB 12
Stratos Idreos
Overview of Data Exploration TechniquesS. Idreos, O. Papaemmanouil, and S. Chaudhuri ACM SIGMOD International Conference on Management of Data, 2015
(explore citations)
DATA SYSTEMSprof. Stratos Idreos
class 19
auto-tuning database kernels
top related