overview and motivation of the icat software suite kerstin kleese van dam
TRANSCRIPT
![Page 1: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/1.jpg)
Overview and Motivation of the ICAT
Software SuiteKerstin Kleese van Dam
![Page 2: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/2.jpg)
Science and Technology Facilities CouncilSTFC employ more than 2200 staff
who are deployed at 7 locations, these are: Swindon where the
headquarter is based, the Rutherford Appleton Laboratory, the Daresbury Laboratory, the Chilbolton Observatory, the UK Astronomy Technology Centre in Edinburgh, the Isaac Newton Group of Telescopes on La Palma; and the Joint Astronomy Centre in Hawaii.
![Page 3: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/3.jpg)
Research and Science Support at STFC
Deliver world class science Engender world class science
Communicate world class science
Annually over 15000 visiting Scientists from around the world from both Academia and Industry.
![Page 4: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/4.jpg)
Why an Integrated e-Infrastructure is required
HPC
HPCAnalysis
Storage
Storage
Analysis
Experiment
ExperimentComputing
HPC
Scientist
![Page 5: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/5.jpg)
What STFC aim to achieve with their e-Infrastructure
Enabling users to get rapid access to their current and past data, related experiments, publications etc., leading to improved analysis through more complete information.
Creating a powerful, long lasting scientific knowledge resource.
![Page 6: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/6.jpg)
Integrated e-Infrastructure
Proposal
Metadata Catalogue
Information
Experiment
Data Acquisition
System
Secure Storage
Data
Analysis
Publication
E-PubsProposal System
All Data and Metadata Capture is automated.
![Page 7: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/7.jpg)
e-Infrastructure – Access to Multiple
Facilities(2)
Data Portal
SNS - ORNL
ISIS – TS1 + 2
DLS
CLF
CSL - Canada
SRS + ERLP
![Page 8: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/8.jpg)
How we achieve the integration
HPC
HPCAnalysis
Storage
Storage
Analysis
Experiment
ExperimentComputing
HPC
Metadata
Scientist
![Page 9: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/9.jpg)
ICAT Software Suite
![Page 10: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/10.jpg)
The ICAT software suite centrally catalogues all experiment related information and extracted key results. Where ever possible information is gathered automatically trough integration with existing IT systems such as proposal systems or data acquisition.The catalogue and the data it references are accessible via a well defined API for easy embedding into any applications.
Distributed Data
Metadata Catalogue
Generic Catalogue Access Interface
Data Access and Analysis Applications
![Page 11: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/11.jpg)
Underlying Data Infrastructure
Online Proposal System
User Office System incl.:
User Database
Scheduling
Health and Safety
Proposal Management
Metadata Catalogue
Data Acquisition System
Storage Management
System
DataAccessPortal
Single Sign On Account Creation and Management
ICAT Software Suite, providing the crucial integration of key functions.
![Page 12: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/12.jpg)
The online proposal system is the entrance point to the Data Management System, and is a rich source of contextual information about the users experiment.
ICAT and the STFC Proposal Systems
![Page 13: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/13.jpg)
ICAT and STFC Data Acquisition
Plug-ins for the data acquisition system ensure automatic, quality controlled collection of data and metadata. ICAT can be easily linked to any existing system.•ISIS :
- SECI (C#, .net) with link to LabView and openGenie•DLS :
- Generic Data Acquisition (Java, on top of EPICS) •CLF :
- For Laser Diagnostics, (LabView)
![Page 14: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/14.jpg)
ICAT and DLS Storage Management
DLS uses the Storage Resource Broker for its Storage Management, this has been integrated with ICAT for data access and delivery.
Main advantage : •Decoupling physical file location from the logical one.•Strict Security•Expandable to many storage systems
SRB MCATDatabase
SRB MCATServer
SRB ADSServer
SRBClient
SRB DiskServer (Local Server)
Atlas Data Store
SRB MCATDatabase
SRB MCATServer
SRB ADSServer
SRBClientSRBClient
SRB DiskServer (Local Server)
SRB DiskServer (Local Server)
Atlas Data StoreAtlas Data Store
![Page 15: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/15.jpg)
ICAT and ISIS Storage Management
ISIS uses their own in house developed data storage access system called Data.ISIS.
Similar to SRB it abstracts from the physical location of the files and delivers the same advantageous in terms of decoupling of logical and physical location of files and security.
![Page 16: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/16.jpg)
ICAT Architecture
Online Proposal System
User Office System incl.:
User Database
Scheduling
Health and Safety
Proposal Management
Metadata Catalogue
Data Acquisition System
Storage Management
System
DataAccessPortal
Single Sign On Account Creation and Management
ICAT Software Suite, providing the crucial integration of key functions.
![Page 17: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/17.jpg)
ICAT 3.3 Aims and Objectives
• ICAT API Version 3.3 aims to be the Grid aware software infrastructure that enables
applications to exploit the capabilities of the ICAT catalogue.
• Data Portal Version 3.3 aims to be the Grid aware software infrastructure that serves the Data Search and Retrieval (DSR) requirements of the STFC. It makes use of the ICAT API 3.3 .
![Page 18: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/18.jpg)
Overall Architecture Principles
• The ICAT software suite has a modular design with clear functional boundaries for each component.• Core functionalities have been grouped together, customisable presentation layers are separated from the function layer to achieve easy maintenance, easy customisation, insulation from changes to underlying areas.• All interaction with the ICAT catalogue are now through the ICAT API.
![Page 19: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/19.jpg)
![Page 20: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/20.jpg)
Core Scientific Metadata
Model
(CSMD)
![Page 21: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/21.jpg)
Rich Data at STFC
Scientific Data of the highest Quality is produced at STFC Facilities and
Departments.The continuity and longevity of STFC has
led to a unique wealth of Information.
How about a system that would give access to all of it independent of where it was produced?
![Page 22: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/22.jpg)
Model Motivation (1)
Most Scientists think in terms of Studies during which they perform a number of investigations e.g. experiments,
observations, measurements and simulations. Results from these investigations usually run through
different stages: raw data, analysed or derived data and end results. Data should be grouped accordingly.
Metadata and Software (e.g. STFC DataPortal) should allow the user to search for interesting data.
Not all information captured in specific metadata schemas e.g. CML, would be used to search for this data or
distinguish one data set from another, give possibility to select special parameter.
![Page 23: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/23.jpg)
Model Motivation (2)
A common general format/standard for Scientific Studies and data holdings metadata did not exist
By proposing Model and Implementation:– Form a specification for the types of metadata
studies should capture during Scientific Studies– Ease citation, collaboration, exploitation and
Integration– Allow easy Integration of distributed
heterogeneous metadata systems into a homogeneous (albeit virtual) Platform
Therefore – The Common Scientific Metadata Model (CSMD) developed.
![Page 24: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/24.jpg)
General Layout
Why – i.e. what was the needWhat is it – description – support keyword searches and taxonmic approaches – data
organisation like a file systems but support linking to a database also
Where is it used – project & softwareWhat are the users likely to search on
What distinguishes one study/investigation/data set from the next
![Page 25: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/25.jpg)
Metadata Model Structure
The Common Scientific metadata model (CSMDM) is a study-data set orientated model holding study information about:
– Topic Indexing– Provenance– Data Holding– Legal notes
• Copyright, patents and conditions of use etc relating to the study and the data in the study
– Related Material• Publications, Community
information and related links– (Access Conditions)
Metadata Granule
Topic
Study
Access Conditions
Related Material
Legal Note
Data Holding
Investigation 1 M
11
Atomic Data Object
Data Collection
11
1
M
M
M
M
1
![Page 26: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/26.jpg)
Model Breakdown: Provenance
The Study contains the following metadata:
– The Study Name– The Study Institution– The Investigator– Extended Study Information
• Abstract• Funding • Start and End times
– Investigations
![Page 27: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/27.jpg)
Investigations
A Study can have more than one investigation; possible
enumerations are experiment, simulation,
measurements etc. – investigations contain:
– Name– Investigation Type– Abstract– Resource– Link to DataHolding
![Page 28: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/28.jpg)
Topic (for indexing)
Keywords– Discipline (i.e. domain)– Keyword Source (e.g.
domain dictionary)– Keyword
Subjects
– Discipline– Subject Source (e.g.
domain taxonomy)– Subject
![Page 29: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/29.jpg)
Access Condition & Related Material
Access Conditions
– Contains a list of users or groups who are allowed access to the metadata and data, or a pointer to an access control system which contains such data for this study
Related Material
– One or many links and or textual descriptions of material related to this study e.g. earlier studies or parallel studies
![Page 30: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/30.jpg)
Data
Data Description holds a logical description of the
Study’s data:
– Data Name– Type of Data– Status– Data Topic– Parameters– Related Data Ref– Relation type (e.g.
derived)
Data Location contains the link between logical name
(e.g. URI's) and physical URLs
– Data Name– Locator(s) (In the
case of Atomic Data Objects these can refer to files as well as named Selects on a database – i.e. virtual data objects)
![Page 31: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/31.jpg)
More on Parameters
Parameters contain a lot of information about the atomic data objects (ADO) and collections
A collection/ADO can have many parameter entries, each parameter entry contains:
Parameter derivation (e.g. measured/fixed)
– The value– The units– Range – Error margin
Parameter aggregation is also supported
![Page 32: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/32.jpg)
Cardinality Issues
The model recommends a certain cardinality of elements
Certain metadata components are necessary for one to have an instance of the implemented model – treating everything as
optional is not acceptableIt is though implementations may modify
this more to their needs – model attempts to remain ideal (i.e. most common Cardinality)
![Page 33: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/33.jpg)
Enumeration Issues
Enumerations (or controlled vocabularies) e.g. types of investigator, types of institutions; these are distinct from the model e.g. as
taxonomies are.However they are necessary for the model to
work so implementations e.g. STFC DataPortal implementation of the model propose some
enumerations for common thingsRecognised and relevant controlled vocabularies are hoped to be used by implementations where
they are available
![Page 34: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/34.jpg)
Conformance Level
For a complete metadata study-dataset record a large amount of metadata has to
be stored/processedSo it’s useful to have conformance levels
Model uses 5 levelsEach level specifies more metadata (and
Indexing information) should be held
![Page 35: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/35.jpg)
Level 1
Type of Information captured:
– Study and Investigation metadata with indexing at the Study level
Level 1 metadata is similar to library/publication style metadata (e.g.
DublinCore)
![Page 36: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/36.jpg)
Level 2
Type of Information captured:
– Level 1 + DataHolding metadata (i.e. DataSets and DataObjects)
![Page 37: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/37.jpg)
Level 3
Type of Information captured:
– Level 2 + related material, Access condition, indexing to data collection levels
![Page 38: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/38.jpg)
Level 4
Type of Information captured:
– Level 3 + indexing to data object level and data object parameter information
![Page 39: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/39.jpg)
Level 5
Type of Information captured:
– All metadata components are filled as L4 + funding, resources used, facilities used etc
![Page 40: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/40.jpg)
Conformance Levels
L1 is similar to library/publication style metadata (e.g. DublinCore)
The current DataPortal uses somewhere between L4 and L5 –the new systems designed with CSMD
conforms to L4+ Benefit of conformance levels; the higher the level of conformance to the CSMD the richer the clients
that operate on the data can be– e.g. identifying datasets and atomic data
objects which link directly to keywords/taxonomies and not just studies
![Page 41: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/41.jpg)
CSMD Used on DataPortal
Implementation used as Data Interface for
DataPortalSingle view of heterogeneous
systems/schemasActs as a stress test of the
model– Limitations feed into
Model Requirements– New requirements
feed back into implementation
![Page 42: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/42.jpg)
ICAT Schema 3.3
![Page 43: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/43.jpg)
![Page 44: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/44.jpg)
Specifics of the ICAT 3.3 Schema
![Page 45: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/45.jpg)
ICAT 3.3 Schema - Facility
![Page 46: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/46.jpg)
ICAT 3.3 Schema - Study
![Page 47: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/47.jpg)
ICAT 3.3 Schema – Study (2)
![Page 48: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/48.jpg)
ICAT 3.3 Schema – Study (3)
Study Investigation
Study Status
![Page 49: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/49.jpg)
ICAT 3.3 Schema - Investigation
![Page 50: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/50.jpg)
ICAT 3.3 Schema - Instrument
![Page 51: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/51.jpg)
ICAT 3.3 Schema - Shift
![Page 52: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/52.jpg)
ICAT 3.3 Schema – Shift (2)
![Page 53: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/53.jpg)
ICAT 3.3 Schema - Keywords
![Page 54: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/54.jpg)
ICAT 3.3 Schema - Topic
![Page 55: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/55.jpg)
ICAT 3.3 Schema – Topic (2)
Topic
Topic List
![Page 56: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/56.jpg)
What is an Ontology?
•Ontologies are used to capture knowledge about a domain of interest.
• An ontology describes the concepts in the domain and the relationships that hold between those concepts.
![Page 57: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/57.jpg)
Advantages of Ontologies
• Provide increased flexibility when representing frequently changing viewpoints of information.
• Alterations can be simply followed up in the model without having to alter the applications on which they are based.
• Allows a unified view of heterogeneous data sources.
• Remove conflicts and terminological uncertainties.
• Facilitate Moderated searches, optimisation of the search results.
![Page 58: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/58.jpg)
Why Ontologies are a useful Solution?
•At present over 1,700,000 keywords describing experiments are housed in ISIS ICAT many of which are synonyms.
•These keywords are used to index experimental studies, however this is seen as a limited method as these free text keywords have no context, and are hard to map by non-experts to terms used by facilities in the same domain and harder still to those outside.
•The creation of ontologies at ISIS will aid in the mapping of concrete manifestations of familiar terms in one domain as well as related concepts in different domains.
•This will facilitate searching of data by category and grouping of data into keywords across studies.
•This could aid in the cross facility searching of related scientific data from the various scientific facilities housed at STFC e.g. CLF and DLS.
![Page 59: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/59.jpg)
A Protégé-OWL Ontology
• Classes• Individuals• Properties
A class is a concept in the domain - a class of People - a class of Pets - a class of Countries
A class is a collection of elements with similar properties.
Instances of classes- America can be an instance of the class Country.
Gemma
Mathew
Fluffy
Italy
America
England
Fido
Class Person
Class Pet
Class Country
livesIn
hasSibling
hasPet
![Page 60: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/60.jpg)
ISIS Facilities Ontology Hierarchy
![Page 61: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/61.jpg)
Class ISISExperiment
Class DataFile
Class Year
wasConductedIn
hasInvestigator
Class Instrument
Class InvestigatorHRP00145.RAW
1986
Pete Jones
HRPD
Class CrystallographyGroupExperiment
hasUsedInstrument
HydraziniumClass InvestigationTitle
hasTitle
hasDataFileName
Protein Crystallography GroupExperiment
ISIS Facilities Ontology
![Page 62: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/62.jpg)
![Page 63: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/63.jpg)
Sample, Investigator and Experiment Ontologies
Sample Investigator Experiment
![Page 64: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/64.jpg)
Ontology Maintainer
•A web application for graphically displaying current versions of an ontology
•Currently ontologies are built within Protégé, an editing environment
•Difficulty in showing constructed ontologies to other domain experts
•The OntoMaintainer allows users to visualize ontology and enter feedback on the classification and structure of the hierarchy
•Encourages collaboration between domain experts (scientists) and ontology builders by allowing members of the community to be involved in the development and maintenance of ontologies
![Page 65: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/65.jpg)
Topic Mapping Tool
•Mapping Tool provides a way of linking proposal system data to the structure of the ontology.
•Data is mapped to the ontology structure according to a set of defined rules.
Proposal System Database
Ontology Mapping Rules
![Page 66: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/66.jpg)
Mapping Tool
![Page 67: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/67.jpg)
Object
Sample Detail
Chemical Formula Name SampleType
LiquidLiquidpoly{1,4-phenylene-[9,9-bis(4-phenoxy butylsulfonate)]fluorene-2,7-diyl} ; C12E5;
D2O
poly{9,9-bis[6-(N,N-trimethyl-ammonium)hexyl]fluorene-co-1,4-phenylene};C12E5;D2=
C37H52N2I2:C22H46=6;D2=
C37H30S2O8; C22H46O6;D2O
![Page 68: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/68.jpg)
•Ontologies would help maximise the value of data collected at ISIS and other STFC facilities by improving the access, navigation and reuse of data.
•Ontologies would facilitate the mapping of terms across STFC facilities which will allow cross-facility searching e.g. external users will be able to search for all experiments carried out across STFC using a powder diffractometer (instrument) even if they do not know the local names of the specific instruments.
•The OntoMaintainer will facilitate the process of creating and maintaining ontologies by providing a means of getting feedback directly from domain experts
![Page 69: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/69.jpg)
ICAT 3.3 Schema - Investigator
![Page 70: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/70.jpg)
ICAT 3.3 Schema – Investigator (2)
Investigator
Facility User
![Page 71: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/71.jpg)
ICAT 3.3 Schema - Sample
![Page 72: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/72.jpg)
ICAT 3.3 Schema – Sample Parameter
![Page 73: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/73.jpg)
ICAT 3.3 Schema – Dataset
![Page 74: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/74.jpg)
ICAT 3.3 Schema – Dataset (2)
![Page 75: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/75.jpg)
ICAT 3.3 Schema – Dataset (3)
![Page 76: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/76.jpg)
ICAT 3.3 Schema – Dataset Status
![Page 77: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/77.jpg)
ICAT 3.3 Schema – Dataset Type
![Page 78: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/78.jpg)
ICAT 3.3 Schema – Dataset Parameter
![Page 79: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/79.jpg)
ICAT 3.3 Schema – Data File
![Page 80: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/80.jpg)
ICAT 3.3 Schema – Data File (1)
![Page 81: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/81.jpg)
ICAT 3.3 Schema – Data File (2)
![Page 82: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/82.jpg)
ICAT 3.3 Schema – Related Data Files
![Page 83: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/83.jpg)
ICAT 3.3 Schema – Data File Parameter
![Page 84: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/84.jpg)
ICAT 3.3 Schema – Authorisation
![Page 85: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/85.jpg)
Other ICAT Related Schema
![Page 86: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/86.jpg)
![Page 87: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/87.jpg)
ICAT API Session Schema
There are 3 tables to the schema, the user, user_session and myproxy_servers:
USER – All users who have logged inUSER_SESSION – All user’s sessions on IcatMYPROXY_SERVERS -- configuration information about which server to logon to
![Page 88: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/88.jpg)
ICAT Core Database Schema
![Page 89: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/89.jpg)
ICAT Core Session
![Page 90: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/90.jpg)
ICAT Core Event
![Page 91: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/91.jpg)
ICAT Core User
![Page 92: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/92.jpg)
ICAT Core DataBase
![Page 93: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/93.jpg)
![Page 94: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/94.jpg)
ICAT Core Database
The core ICAT catalogue is at STFC run on an Oracle 10G RAC clustered database server.
The system has been customised to make efficient use of the offered features of Oracle.
If required these could however be removed in the future.
![Page 95: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/95.jpg)
ICAT API
![Page 96: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/96.jpg)
ICAT Architecture
Online Proposal System
User Office System incl.:
User Database
Scheduling
Health and Safety
Proposal Management
Metadata Catalogue
Data Acquisition System
Storage Management
System
DataAccessPortal
Single Sign On Account Creation and Management
ICAT Software Suite, providing the crucial integration of key functions.
![Page 97: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/97.jpg)
![Page 98: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/98.jpg)
ICAT API Version 3.3 (1)
The ICAT API version 3.3 is the interface that any application should use to interact
with the core ICAT system catalogue. At present it is used by applications such as the ISIS XML ingest, the DLS Generic Data
Acquisition System, DLS DDH and the DataPortal. The API offers a wide range of web services for the easy interaction with
the ICAT core catalogue.
![Page 99: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/99.jpg)
ICAT API Version 3.3 (2)
The ICAT API version 3.3 consists of three main components:
•Web Services offered to other applications•ICAT Catalogue Interactions•ICAT Catalogue Session Management
![Page 100: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/100.jpg)
ICAT API Version 3.3 (3)
The ICAT API version 3.3 uses JPL and SQL to directly interact with the underlying oracle databases
The ICAT API version 3.3 has been written in Java using EJB3, JPA and JAX-WS
![Page 101: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/101.jpg)
ICAT API Version 3.3 (4)
Web Services offered to other applications for the Search, List, Ingest, Delete, Modification of:
AuthenticationInvestigation, Datafile and Dataset InformationInvestigatorKeywordsPublicationSampleDownload
![Page 102: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/102.jpg)
DataPortal
![Page 103: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/103.jpg)
ICAT Architecture
Online Proposal System
User Office System incl.:
User Database
Scheduling
Health and Safety
Proposal Management
Metadata Catalogue
Data Acquisition System
Storage Management
System
DataAccessPortal
Single Sign On Account Creation and Management
ICAT Software Suite, providing the crucial integration of key functions.
![Page 104: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/104.jpg)
![Page 105: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/105.jpg)
DataPortal for ICAT Version 3.3
The DataPortal is a highly customisable web interface to interact with the ICAT version 3.3.There are at present two distinctive versions one for ISIS and one for DLS. Whereas the underlying functionality is the same the graphical representation and choice of used services varies.The DataPortal offers a number of search interfaces, the ability to explore investigations and download associated data.
![Page 106: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/106.jpg)
![Page 107: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/107.jpg)
![Page 108: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/108.jpg)
![Page 109: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/109.jpg)
![Page 110: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/110.jpg)
![Page 111: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/111.jpg)
![Page 112: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/112.jpg)
![Page 113: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/113.jpg)
![Page 114: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/114.jpg)
![Page 115: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/115.jpg)
![Page 116: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/116.jpg)
![Page 117: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/117.jpg)
Top Left Hand Menu
![Page 118: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/118.jpg)
Bottom Left Hand Menu
![Page 119: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/119.jpg)
Top Right Hand Menu
![Page 120: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/120.jpg)
Session Expire
![Page 121: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/121.jpg)
DLS DataPortal
![Page 122: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/122.jpg)
![Page 123: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/123.jpg)
![Page 124: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/124.jpg)
![Page 125: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/125.jpg)
![Page 126: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/126.jpg)
![Page 127: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/127.jpg)
![Page 128: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/128.jpg)
![Page 129: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/129.jpg)
![Page 130: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/130.jpg)
![Page 131: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/131.jpg)
![Page 132: Overview and Motivation of the ICAT Software Suite Kerstin Kleese van Dam](https://reader038.vdocuments.us/reader038/viewer/2022110213/5697bf921a28abf838c8f04d/html5/thumbnails/132.jpg)
Questions?