(ATS4-DEV02) Accelrys Query Service: Technology and Tools

Jim Barstow

Research Fellow, R & D

Jim.barstow@accelrys.com

The information on the roadmap and future software development efforts are intended to outline general product direction and should not be relied on in making a purchasing decision.

Agenda

• Challenges with the current data environment

• Quick look at commonly used tools

• What is the Accelrys Query Service?

– How does it address these challenges?

– What tools are available?

– How is it used?

• Short demos but the next session (ATS04-DEV03) is all demos

Challenges with today’s data environment

• Data models have become very complex and very dynamic

– A big data model used to have 10’s of tables, that’s now small

– Assay data models huge and may expand frequently

– Data models may differ by user or project

• The physical data model is usually optimized for storage

– Can be very unfriendly to consumers of the data

– Tall and skinny is a common paradigm

• How do you deliver such complex data to the scientist in a way they can understand and utilize?

– Need a logical data model that is usually very different from the physical model

– This logical model needs to be easy to maintain and update

What is the solution?

• Middleware that can provide an easy to use logical data model that is mapped to the physical model

– Make the data easy to query and understand by scientists

– Adapt to dynamic data models rapidly (automatically)

– Expose subsets of the data model as necessary

– Ideally, no new warehouse required • Use the data in place

Existing Solutions – Warehousing tools

• There are a variety of these available

• Pros – Does make the data more accessible

– Provides an opportunity to sanitize and transform the data

• Cons – Introduces another database to maintain

– The “logical” data model is still really a physical model

– Flexible and dynamic data models still a problem

– Same problem, just simpler

• Going on 20 years old • Pros

– Provides a way of building a single static hierarchical logical data model on top of physical model

– Works well with medium complexity data models – End to end solution – Easy for end users

• Cons – Difficult to handle the most complex models – No capacity for handling dynamic models – No built in way of transforming data (e.g., pivoting) – Application development is challenging – Dated, no thin client support

Existing solutions - ISIS

ISIS Query Builder

• Non-warehouse based logical to physical mapping • Pros

– Provides a much more sophisticated logical model than ISIS – Handles the most complex data models

• Used for data models with 1000’s of tables • No warehouse required, uses data in place

– Built in support for data transformation (e.g., pivoting) – Multiple solutions for dynamic data models – Builds on and enhances the familiar relational environment – Fully supported in Pipeline Pilot and via standard RESTful interfaces

• Cons – It’s new

Accelrys Query Service (AQS)

• The logical data model is more than just a single hierarchy – It is a network of inter-related data

– This network can be interpreted as a set of related hierarchies

– Each hierarchy provides a different logical view

• The logical data model may be dynamic – Created on-the-fly from metadata

– Customized by a user’s role or area of interest

What is the AQS philosophy?

Molecule Products Catalog Supplier

Data Network

Catalog

Supplier

Molecule

Products Molecule

Products

Catalog

Supplier Molecule

Products

Catalog

Supplier

Derived Hierarchical Relationships

Products Hierarchy Molecule Hierarchy Catalog Hierarchy

Data Networks and Derived Hierarchies

• By an admin using Data Source Builder (demo) • Each node in the network defines a source of data

– Typically a table but • A Pipeline Pilot protocol • A web service • Another network

• Each node has relationships to other nodes – These relationships define the shape of the network – Relationship specified by a “connector” object

• Some connectors are defined as gateways to the network – These become the roots of the derived hierarchies

How is this network defined?

What is a “connector”?

• Specifies how 2 sources of data are connected

• Key field relationships

• Specifies multiplicity (1:1 vs 1:many)

• Can transform the data – Pivot

• Can filter the data by fixed criteria

• Controls how the connected data is exposed

• Just a (relatively) simple XML element – Easy to create in on-the-fly

• The Query Service is a RESTful web service running in Tomcat in the AEP

• It can be accessed via

– A rich set of Pipeline Pilot components in the Query Service Collection

– Standard RESTful calls from any system that supports web access

– The different approaches have the same AQS functionality • There is no “backdoor”

How do I access the Query Service?

Demo

Complex data models in Data Source Builder

How are complex / dynamic physical models handled?

• Create all or part of the network at startup – Dynamic pivots

– Bootstrap from metadata

• Prune connectors based on permissions or role

• Expand or contract the network on-the-fly – User selection of areas of interest

– Add or remove subnetworks

• Lots of experience in the field

• Demo in the next section

• The Accelrys Query Service will be available in 9.0

• Provides a way of building easy to use logical data models over your physical data model

• Makes it much easier to handle complex and dynamic data models

• Building a network from scratch in the next section

Summary