data structures and other

7/29/2019 Data structures and other

1/72

Data structuresExamples

Data Structures and beyond

Igor Petruk

EPAM Systems.

London, 2013

Igor Petruk Data Structures and beyond


2/72


i

1 Data structuresGoals

What is a data structure?

2 Examples



3/72


GoalsWhat is a data structure?



4/72



Goals of the presentation

Find data structures you like (maybe reminds a part of your system)


G


5/72





Make a note with their names


D t t t G l


6/72






Become aware of availability of the trickiest data structures


Data structures Goals


7/72







Discover some cool ways to use them




8/72







Discover some cool ways to use themGet inspired to practice data structure driven development




9/72







Discover some cool ways to use themGet inspired to practice data structure driven development

Not a goal of the presentation

Trying to remember what I say




10/72




data structure is a particular way of storing and organizing data in acomputer so that it can be used efficiently.




11/72

Examples What is a data structure?



Built of

DataBehaviour




12/72




Built of

DataBehaviour

They can


Data structuresE l

GoalsWh i d ?


13/72




Built of

DataBehaviour

They can

Be hard to imagine

Be complex to understandSolve your problem in one go


Data structuresE l

GoalsWh t i d t t t ?


14/72


What kinds are out there?

Some examples (from Java)

HashMapLinkedList

Array

TreeSet

Anything in common?





15/72




HashMapLinkedList

Array

TreeSet

Anything in common? Eager, finite, mutable





16/72




HashMapLinkedList

Array

TreeSet


They can be

Eager vs Lazy





17/72




HashMapLinkedList

Array

TreeSet


They can be

Eager vs Lazy

Mutable vs Immutable





18/72

p



HashMapLinkedList

Array

TreeSet


They can be

Eager vs Lazy


Persistent





19/72

p



HashMapLinkedList

Array

TreeSet


They can be

Eager vs Lazy


PersistentFinite vs Infinite





20/72



HashMapLinkedList

Array

TreeSet


They can be

Eager vs Lazy


PersistentFinite vs InfiniteComputed





21/72



HashMapLinkedList

Array

TreeSet


They can be

Eager vs Lazy



Probabilistic





22/72



HashMapLinkedList

Array

TreeSet


They can be

Eager vs Lazy



Probabilistic

Optimized for specific environmentOn disk




23/72

MultiSet

What is it

A set

Each value contains a counter




24/72

MultiSet

What is it

A set


Usage

StatisticsPut values inGet element occurence count




25/72

MultiSet

What is it

A set


Usage

Statistics

Put values inGet element occurence count

Save memoryStill using a set




26/72

MultiSet

What is it

A set


Usage

Statistics



Uniqueness sanity checkPut values in

Filter entries with count>1Collect your errors




27/72

MultiSet

What is it

A set


Usage

Statistics



Uniqueness sanity checkPut values in

Filter entries with count>1Collect your errors

Maintain leadersPut values in realtimeUse max-first iterator



fi


28/72

Trie - Prefix tree

What is it



T i P fi


29/72

Trie - Prefix tree

Branch keys can be anything

StringChars

Custom objects

BitsBit chunks of hash, Hash Array Mapped Trie



T i P fi t


30/72

Trie - Prefix tree


StringChars

Custom objects


Usage

As a mapNo collisions with linear search in bucketNo rebalancing



T i P fi t


31/72

Trie - Prefix tree


StringChars

Custom objects


Usage


Prefix searchBest support for this operation



T i P fi t


32/72

Trie - Prefix tree


StringChars

Custom objects


Usage



CompressionIf prefixes are shared



Trie Prefix tree


33/72

Trie - Prefix tree


StringChars

Custom objects


Usage



CompressionIf prefixes are shared

Indexing, Full Text SearchSuffix Trie



Splay tree


34/72

Splay tree

Splay treeBalances in order last access first



Splay tree


35/72

Splay tree


UsageLRU Cache



Splay tree


36/72

Splay tree


UsageLRU Cache

Why not to talk about cache in general?



Cache


37/72

Cache

Cache

Cache is not a BIG THING

Caches that are as lightweight and simple as a map are available(Guava)



Cache


38/72

Cache

Cache

Cache is not a BIG THING

Caches that are as lightweight and simple as a map are available(Guava)

Data

Simple trees, splay trees

Databases

Files

Off-heap



Cache


39/72

Cache

Behaviour

Population strategyManualLoading



Cache


40/72

Behaviour

Population strategyManualLoading

Write strategyRead-onlyWrite throughWrite behind



Cache


41/72

BehaviourPopulation strategy

ManualLoading

Write strategyRead-onlyWrite throughWrite behind

Eviction strategyLRUMRULFU

RandomGarbage collector drivenMulti Queue



Cache


42/72

Complex?



Cache


43/72

Complex?

LoadingCache graphs = CacheBuilder.newBuilder().maximumSize(1000).expireAfterWrite(10, TimeUnit.MINUTES)

.build(new CacheLoader() {

public Graph load(Key key) throws AnyException {return createExpensiveGraph(key);

}});



Cache


44/72

Usage other then obvious

Protecting against eventual consistency issues

Batching

Memoization

Dynamic programming

Concurrent synchronization



View


45/72

View

Is this a data structure?

YES/NO



View


46/72

View

Is this a data structure?

YES/NO

We have the data (somewhere)

We have the behaviour

Why copy?



View


47/72

Usage

See a list as a map



View


48/72

Usage

See a list as a map

Process only needed part of data



View


49/72

Usage

See a list as a map


Perform multiple operations togetherEach operation produces a wrapper

Final operation - copy



View


50/72

Usage

See a list as a map



Final operation - copyDoes anyone know how to concat two immutable lists with O(1)complexity?


Data structures

Examples

View


51/72

Usage

See a list as a map




Combine data structures


Data structures

Examples

View


52/72

Usage

See a list as a map




Combine data structures

Use listeners to build change aware viewIndexing

Live displayReactive programming


Data structures

Examples

Persistent data structures


53/72


Any mutation produces new version of a structure

Previous version remains unless deleted (or garbage collected)

Usually designed to reuse most of the previous version


Data structures

Examples



54/72





Examples

Immutable linked list (cons list)

Immutable trees

Usually immutable functional data structure are persistent


Data structures

Examples



55/72





Examples


Immutable trees


Usage

When we have a single writer - they are lock free


Data structures

Examples



56/72





Examples


Immutable trees


Usage

When we have a single writer - they are lock freeAny instance of PDS is a snapshot already


Data structures

Examples



57/72





Examples


Immutable trees


Usage

When we have a single writer - they are lock freeAny instance of PDS is a snapshot already

Put a journal in front of writer, store PDS to disk sometimes and hereit goes - durable high performance in-memory database with O(1)snapshot support


Data structures

Examples

Bloom filter


58/72

Bloom filter

A probabilistic set

Can represent set of elements from infinite universe


Data structures

Examples

Bloom filter


59/72

Bloom filter

A probabilistic set


Strict test if the set does NOT contain the element


Data structures

Examples

Bloom filter


60/72

Bloom filter

A probabilistic set



Probabilistic test if the set contains the element


Data structures

Examples

Bloom filter


61/72

Bloom filter

A probabilistic set




Internals

An array of m bits

k different hash functions, that produce values from 0 to m-1

Extendable to counting Bloom filter which supports deletion


Data structures

Examples

Bloom filter


62/72

Bloom filter

A probabilistic set




Internals

An array of m bits

k different hash functions, that produce values from 0 to m-1

Extendable to counting Bloom filter which supports deletion

Notable usagesApache Cassandra and Google BigTable

Test if the record can be on the diskIf it can - fetch it from the diskIf not, then we dont have to trigger a disk operation


Data structures

Examples

Stream


63/72

Stream

List

Evaluator in tail

Can be infinite


Data structures

Examples

Stream


64/72

Stream

List

Evaluator in tail

Can be infinite

Usage

Good abstraction for infinite computation that can be consumed by listprocessing routines


Data structures

Examples

Stream


65/72

Stream

List

Evaluator in tail

Can be infinite

Usage


If internal list is a linked list, then the GC can gather alreadyprocessed data


Data structures

Examples

Stream


66/72

Stream

List

Evaluator in tail

Can be infinite

Usage



Change definition of int fib(int) to Stream fib()


Data structures

Examples

Stream


67/72

Stream

List

Evaluator in tail

Can be infinite

Usage




You still can represent input, but via generic Stream


Data structures

Examples

Stream


68/72

Stream

List

Evaluator in tail

Can be infinite

Usage




You still can represent input, but via generic Stream

Convert one stream to another via wrappers


Data structures

Examples

Option


69/72

Option

One or zero element collection


Data structures

Examples

Option


70/72

Option


Null is no more!

JK, easy conversion from null to Option and back


Data structures

Examples

Option


71/72

Option


Null is no more!


Forces you to unbox the value before usage

Usage


Data structures

Examples

Option


72/72

Option


Null is no more!


Forces you to unbox the value before usage

Usage

Do you have and element or not?

Is the computation sucessful or not?

Is the value populated or not by user?

Null safe operations within OptionAvailable in Guava as Optional

Monad


data structures and other

Documents