gemstone collection views

8/7/2019 GemStone Collection Views

1/35

GemStone Collection ViewsGemStone Collection Viewsfor Instant Display of GS Collectionsfor Instant Display of GS Collections

by Paul Baumann

IntercontinentalExchange, Inc.2011.03.15


2/35

Just because...

To share ongoing work toward addressingcommon performance problems.

To show techniques that others may findinteresting.

To encourage developers to think outsidethe box.

It was fun to develop.


3/35

Installation Sources

Efficient GemStone Enumeration

http://techsupport.gemstone.com/forums

http://www.cincomsmalltalk.com/CincomSmalltalkWiki/PostgreSQL+Access+Page

Public Store Repository


4/35

Prerequisites

Efficient GemStone Enumeration base code

http://www.visualworksforums.org/index.php?topic=35.0

- OR -

http://techsupport.gemstone.com/forums/37605-gss-tips-tricks

- OR -

Store Public Repository packageGS_CollectionPerformanceBase

- FROM -


5/35

Download files from the GSS Tips & Trickssection of the GemStone forums website:http://techsupport.gemstone.com/forums

Load the parcel named VW_CollectionViews.

Use a topaz session to file-inGS_CollectionViews.gs as SystemUser.

Detailed instructions in the methodBcvCollectionViews class>> #setupInstructions.

Installation Option 1
http://techsupport.gemstone.com/forumshttp://techsupport.gemstone.com/forumshttp://techsupport.gemstone.com/forums


6/35

Use a VisualWorks image to connect tothe Cincom Public Store Repository

Load GemKit GemStone Source Code Management.

Load PlbCollectionViews.

Detailed instructions in the method

BcvCollectionViews class>> #setupInstructions.

Installation Option 2 Public Store Repository


7/35

Session Configuration

Session login parameters must be configuredto connect the Collection View classes.

CollectionViews.BcvCollectionView class

>>setupInstructions_classConnectors


8/35


9/35

Why Use Collection Views?

An efficient way to display attribute valuesfrom a large GS collection.

An efficient way to #collect: a set of resultobjects from a large GS collection.

An efficient way to replicate results to aGBS client image.

Similar use and benefit as a table View of arelational database.


10/35

Relational Database View

In database theory, a view consists of a stored query accessible as a virtualvirtualtabletable composed of the result set of a queryresult set of a query. Unlike ordinary tables (basetables) in a relational database, a view does not form part of the physicalschema: it is a dynamic, virtual table computed or collated from data in theit is a dynamic, virtual table computed or collated from data in thedatabasedatabase. Changing the data in a table alters the data shown in subsequentsubsequentinvocationsinvocations of the view.

Views can provide advantages over tables:

* Views can represent a subset of the datasubset of the data contained in a table* Views can join and simplify multiple tablessimplify multiple tables into a single virtual table* Views can act as aggregated tablesaggregated tables, where the database engine aggregatesdata (sum, average etc) and presents the calculated results as part of thedata

* Views can hide the complexityhide the complexity of data; for example a view could appear as

Sales2000 or Sales2001, transparently partitioning the actual underlyingtable* Views take very little spacevery little space to store; the database contains only thedefinition of a view, not a copy of all the data it presents

* Depending on the SQL engine used, views can provide extra securityextra security* Views can limit exposurelimit exposure of a table or tables to the outer world


11/35

SQL Comparison

SQL ViewSELECT

name,

money_received,

money_sent,

(money_received - money_sent)AS balance,

address

FROM table_customers c

JOIN accounts_table a

ON a.customerid = c.customer_id

GS ViewAccount>>balance

^money_received - money_sent

Customer>>account

^Accounts at: customer_id

^account

Customers view: #(

name

(account money_received)

(account money_sent)

(account balance)address

)


12/35

SQL View vs. 1.0 GS View

JOIN is achieved through unary messages through theobject graph.

OUTER JOIN is not part of 1.0.

GROUP BY support is not complete in 1.0.

WHERE is achieved with a separate #select:statement.

GS Views can allow updates, but SQL views also allowINSERT and DELETE.

SQL result functions like AVG, COUNT, MAX, MIN, SUM

are not part of 1.0.

1.0 GS Views can only retrieve attribute values from asequence of unary messages (not binary orkeyword).


13/35

1.0 Objectives

Quickly replicate some attributes of a large GScollection to a client image for display.

Minimize the time spent gathering attribute values.

Reduce server paging costs.

Avoid creating an object in GS for each result row.

Return data to client as needed (in chunks of rows).

Avoid replicating entire graphs of objects just to returnsome attribute values.

Easily control replication without replication specs.


14/35

GS View 1.0 Syntax

aCollection

[by: chunkSize]

view: #([replicationLevels] [attributeAlias @] attributeGetter ...)

chunkSize declares the number of rows

that are replicated in each clienttraversal. 0 disables chunking.

replicationLevels is an integer 0-6 (defaultis 1) that controls the depth that anattribute is replicated to a clientimage. 0 avoids replicating the object.

attributeAlias is a symbol that a row

attribute can be known as; the defaultis the last symbol of theattributeGetter.

attributeGetter is either a symbol or anarray of symbols that will be sent toeach object.

EXAMPLE query from GBS client image:

GBSM evaluate: (Globals select: [:ea | ea isBehavior ])

by: 50 view: #(

name

2 instVarNames

2 instSelectors @ selectors

2 classVarNames2 classSelectors @ (class selectors)

0 yourself

)'.


15/35

Traditional Result Sets

Collect to indexed arrays

GBSM evaluate:

(Globals select: [:ea | ea isBehavior])

collect: [:ea |

#[ea name,

ea instVarNames,ea selectors,

ea classVarNames,

ea class selectors,

ea]

].'.

Collect to keyed collection

GBSM evaluate:

(Globals select: [:ea | ea isBehavior ])

collect: [:ea |

Dictionary new

at: #name put: (ea name);

at: #instVarNames

put: (ea instVarNames);

at: #selectors put: (ea selectors);

at: #classVarNames

put: (ea classVarNames);

at: #classSelectors

put: (ea class selectors);

at: #yourself put: ea;

yourself

].'.

Collect to row objects

GBSM evaluate: (Globals select: [:ea | ea isBehavior ])collect: [:ea |

ClassQueryRow newForClass: ea].'.


16/35


17/35

GS View Row

Has all the advantages of the three TraditionalResult Set styles.

Has distributed workload and chunked replication.

Avoids cost of creating a GS object for each rowattribute, yet still preserves row identity inclient image.

Reduced client memory footprint (only one chunk

at a time plus any individual rows stronglyreferenced).

Rows attributes can be refreshed to reflect thecurrent object state.


18/35

Rows are Special

Views are composed of an index foreach attribute--not row instances.

Row instances are only created on-

demand in GS or Client. Identity is preserved for rows created in

the client. Equality is preserved for rows created in

GS. This is the most efficient design for how

views are intended to be used.


19/35

Better Replication

You can specify a view-specific faultlevel for each attribute.

Fewer class connectors are needed

You can return attributes through agraph of GS objects that lack aclient representation (like class andconnector).


20/35

Attribute Replication Levels

Customers view: #(name

(account money_received)

(account money_sent)

(account balance)

address

)

Customers view: #(1 name

1 (account money_received)

1 (account money_sent)

1 (account balance)

0 address

)

0 replicationLevels leaves the object in GStypically a stub in VW.

Used to keep a row associated with objects

without replicating them.

If an attribute value is already client-replicatedthen it stays replicated.


21/35

Attributes Values are a Snapshot

Attribute values do not automaticallyrefresh when the GS object they camefrom does.

Can be useful if you want to see and edit aconsistent view of data regardless ofsubsequent GS changes.

You can refresh attribute values.

You can compare attribute values withcurrent object state.


22/35

Views Can Be Refreshed

Requires #yourself as attribute. #refresh only does something if

#yourself is an attribute.

#refreshAttributes warns if #yourselfis not an attribute.

Number of rows does not (normally)

change.

aView refresh

aView refreshAttributes


23/35

Rows Can Apply Changes

Requires #yourself as attribute.Row change goes to both row

attributes and to object.If you abort object changes then

attribute changes endure (until you#refresh).

row name: 'Fred'

row asForwarder name: 'Fred'


24/35


25/35

AdamFred

ChelseaSusanTrish

AdamFred

ChelseaSusanTrish

TroyEdmundTyrelOliverBret

AdamFred

ChelseaSusanTrish

$300$600

$1002$192$4711

$422$1919$6114$1201$592

$receivednameAdamFred

ChelseaSusanTrish

$200$550$902$192$4711

$400$1919$6114$1201$500

$sentAdamFred

ChelseaSusanTrish

$100$50$100

$0$0

$22$0$0

$1201$92

$balanceAdamFred

ChelseaSusanTrish

anAddressanAddressanAddressanAddressanAddress

address


RicoWarrenHelterHaleyMattox

$192$5992$1843$1119$691

$192$4992$1843$1119$691

$0$1000

$0$0$0


View-Attribute-Chunk Structure


26/35

Chunking How it Works

1.A view that is created will pause after the first chunk ofrows is processed. A suspended GS process waits toresume chunk gathering.

2.First chunk is replicated to client along with view.3.Client automatically forks a process that asks GS to

resume chunk gathering.

4.Client is able to use the first chunk while GS operates inparallel to gather remaining chunks.

5.When client asks for a row that is not replicated (orcached) then it asks GS for the chunk for that row.


27/35

ClientWindowProcess

Server/GemDefaultProcess

Create View

Fork Process

Server/GemGatheringProcess

ClientGatheringProcess

Gather chunkthen wait

Next chunkuntil complete

Fork Process

Show View

Gather chunkthen waitUse active

chunk

Needsdifferent

chunkActivatechunk

ChunkGathered?

Request

Gather specificchunk then wait

Use activechunk

yield


28/35

Chunking Replication Efficiency

Client image transports one chunk of rowattributes at a time.

Fewer objects are contained in the GBS

cache and the GS export set. Less memory is used.

User experiences less of a delay.

Faster when GS takes time gatheringattribute values AND client has somedelay in using all the atttribute values


29/35


30/35


31/35

Chunking Not Always Good

Client referenced only one chunk at atime.

Not good when client immediatelyneeds other chunks.

Not good for rapid random rowaccess on clientlike sorting.

Has more traversal costs (if you useall rows).


32/35

Views on Non-Sequenceable Collections

Contents is copied to a sequenceablecollection as view is created.

View retains no knowledge of theoriginal collection.

The copying of contents to asequenceable collection is thebiggest performance hit that viewscan experience (entirely due to diskpage reads).


33/35

Views on Sequenceable Collections

Much faster than non-sequenceable collections. A view on a sequenceable collection avoids the

page read cost of copying contents. The collection you created the view for is

enumerated to gather attribute values. Do not create a chunked view of a sequenceable

collection that contains objects that changeposition. Changes could affect backgroundchunk gathering.

Workaround options: Avoid chunking for that view.

Create a view for a copy of the collection. Send #gathering_waitForRemaining before

changing the collection.


34/35


35/35

Future Directions

Attribute change reporting.

View sorting.

Outer Joins.

*companies Attributes from functions.

balance $(money_received - money_sent)

Persistent managed shared views.A new kind multi-indexed RC collection

Use of attribute relation grouping.row group: #(marketType product)

gemstone collection views

Documents