25/07/12

Input from Taverna team – slides by Shoaib Sufi

Taverna – State of the nation

Intro• Taverna tools used across project

• Two needs

• Delivery against commitments

• Innovation and new markets (users, proposals, research opportunities)

• Focus on Features

• Focus on Products

• Only touch on the engineering and collab work needed to get the work done

Sources of input• EU Projects

• WF4Ever

• SCAPE

• BioVel

• Helio

• RCUK & TSB

• CA4LS

• Community

• Collaborators

WF4ever• Prov-o export from Taverna

• V1 end July 2012

• V2 October 2012

• Prov-o extensions to make it more workflow centric

• October 2012

• Auto WF rerun

• September 2012 (+2M)

• October 2012 Taverna 2.x server modified to allow Prov-o export (File and API)

• Need: components – annotation/visualise SCUFL2 (API)

• Will aid in detecting service ‘similarity’ for suggesting repair

• This can be incorporated any time (there are other steps to do first) maybe early 2013

• All working being done on SCAPE and components (David/Alan/Finn) applicable

WF4EVER(2)• Need: Taverna 3

• To keep up to date

• Data Bundle

• January 2013 support will be developed in Taverna by wf4ever

• Astro Taverna – perspective and activities plugin

• V1 already

• V1.1 Sept

• v2 December (Hopefully using Taverna Structured Object Support)

WF4ever(3)• Need: More standard Structured Data Support (not just specific VO Table)

• We can currently render VOTable

• Need to be able to input VOTable

• Currently this is Astronomy specific

• E.g. SAMP protocol using RPC on localhost

• Raven helped us use their mess of a java source which had been cut and paste from java classes from multiple OSS projects (OGSi would be able to do the same)

SCAPE• Taverna Components support in 2.x (1st Version end of September 2012)

• Needed by BioVel, WF4Ever and others who want components

• Via a 2.4 plugin

• Workflow annotation (semantic)

• RightField based interface

• Component Profile Spec

• Akin to families

• Plan to use wfdesc from WF4Ever

• Component Validation

• Component Discovery (myExp) changes.

Biovel• Taverna 2.x work end September 2012

• Securing the interaction service

• Grey/whitebox provenance collation

• SCAPE requirement for whitebox

• Greybox – e.g. google refine tells you which steps it has done

• WebDav support

• Component work

• Test harness component

• Service to component (r-click)

• Upgrade and downgrade component versions

• OAuth2.0 support

• BioVel users want to login via facebook/gmail etc not Certificates !!

• ‘Baby portal’ / demo interface / VM

BioVel(2)• Taverna 3 work – features by April 2013

• Collation of notifications

• Annotation of data

• Tracking source/license through the workflow

• Help answer questions such as ‘Where is GBIF data used’

• One button publish of a run

• Include import of a run

• Replay & re-run from cache

• Prettier REST service support

• Structured data i.e. JSON support

• Pluggable server (in 2.x tend to be baked in)

• Taverna lite ongoing support

ca4ls• Ruby Client Libs

• Mostly maintenance; there is an API stable version that has had ‘000s of downloads from RubyGems

• Java Client Libs – Release in August 2012

• Ruby ones was first. Work needed to get this one in to the same state

• To support the new REST API and Secure WF’s

• Need by Tav3 platform using Tav3 sever as an executor

• Being used by WF4Ever, BioVel

• More Server bugs found so need a Server 2.4.1 release

Ca4ls(2)• Updated AMI for Tav Server – Aug 2012 (depends on updated 2.4.1 server / Donal availability)

• Need an official Announcement

• WIBNI – updated demonstrator interface that supports the interaction service

• Interaction Service and Elastic View – Oct/Nov 2012

• 2.4.1 server should bundle in support for the interaction service

• Iteration with Helen and Mike around the NHS workflow

• Incorporate this into Eagle Elastic view

• Cloud Storage Handling

• 3rd party agent initiated data transfers (i.e. passing data to a service from S3 without data going through Taverna and without the service realising the data came from S3)

• WIBNI -> need – will know Oct/Nov 2012

Reflection – things we do right

•Enabling people to connect many resource types together (KW)

•Drag and drop workflow design (KW)

•myExperiment link- (KW,ALB,MR (incl. T2 use in Galaxy, BioCat in MyE)

•Client and server for different types of users/deployments (KW)

•Keeping everything open source (KW)

•Ability to build up a user specific library of services (ALB)

•Support for user developed plugins (ALB)

•Pragmatic REST service support (AW)

•Training/workshops (AW)

•Workflows behind portals to support Science (AW)

2• BioCatalogue in Taverna (MR)

• Impressive amount of functionality in Taverna (MR)

• Usability (MP (PANACEA))

• Taverna support and community – helpful/responsive (MP)

• Updated and maintained – annual new versions (MP)

• Impressive interoperability – JAX-WS, Axis1, Axis2 & diff encodings (UTF8 etc) (MP)

• Tool service – make use of existing user assets/scripts (AW)

Reflection – areas of improvement• Exposing shims (KW)

• Lack of Components (KW)

• Lack of service sets (services+shims) for domainst2 (KW)

• Not exposing enough annotation about the services through the client (KW)

• Annotating individual properties of the service interface, instead of

functional collections of the properties (KW)

• Having no easy way to explore provenance information and results for

past runs (although Daniele's logbook was almost there) (KW)

• Not managing tables of data for inputs/outputs (KW)

• library of services does not include REST services (ALB)

• no partial re-run of workflows (it is usually the beanshell after the

• resource intensive service which fails) (ALB)

• Not domain neutral (you can't disable everything bio-informatic

• specific, e.g. within preferences) (ALB)

2• Bespoke when we should have used off-the-shelf - RAVEN (AW)

• Provenance not driven by user problems (AW)

• Getting rid of Ian’s and Stians data handler (AW)

• Annotating = science notes = good science not emphasized enough (MR)

• Annotation should be in your face and fun – it’s not (MR)

• Unstructured annotations in Taverna and BioCat (MR)

• Functionality over ease of use – although it is best to be able to do something even if hard (MR)

• No incentives for making high quality workflows (MR)

• 10 best practices of workflows not always easy to do with current tooling (MR)

• Need more examples, training and interfaces (CAG)

3• MP (PANACEA)

• Design of a generic Interface for asynchronous web services with a single component box like it is with Soaplab web services. (MP)

• Improve robustness and large data handling: (T 2.4 has been a really good improvement), reduce memory footprint, there are some problems when using direct data instead of referenced data when processing large amounts of data or a lot of files with read/write operations, etc. (MP)

• Provenance:

• the progress report should be exportable, (MP)

• the user should be able to configure the provenance (the user would like to choose which data should be stored), etc. (MP)

• Input interface local service: It would be interesting to have a predefined component that could be put at the beginning of every workflow that could be configured for every execution to read data from a folder, or from a list of urls, or from a text box etc. Now, we need to have 2 or 3 different versions of the same workflow just because in some cases we process lists of urls or files in a local folder. I understand this could be a very particular scenario but adding new and improved local services helps a lot. (MP)

• *Taverna Server installation and easy to use web interface for users (MP)

Taverna 3 plan• Workbench Beta Feb 2013

• Products:

• Taverna Server (infrastructure)

• Taverna Workbench (user interface)

• Taverna Command Line Tool (flexible running)

• Demonstrator interface (making infrastructure usable)

• Taverna Platform (embedding) – Dec 2012 / Jan 2013

• SCUFL2 & Tools (WF formats and research objects)

• Server client libraries (easy to script the server)

Taverna 3 Features• Components (porting of work done for SCAPE) + components view

• Iteration visualization (indication in diagram, "what if" + more TBD)

• SCUFL2 support (reading, writing, running)

• Research Object support

• In OSGi

• Structured data (tables, arrays and beans)

Taverna 3 Features (2)• Grey box provenance

• Remote enactment from the Workbench (using the server from the workbench)

• Customization (service sets, plugins)

• Java Scripting Framework service - subsumes beanshell and perhaps others

• (optional) SPARQL - needs significant work

• Interaction service

• OSGi update mechanism

• Documentation: Plugin developers migration guide (very important)

• Design Objective: maintain look of Workflows diagram (but with enhancements e.g. visualisation of iterators)

Taverna 3 subsystems• Beta by Feb 2013

• Convert Workbench UI modules to OSGi

• Update Run perspective

• myExperiment & BioCatalogue perspectives stay as is by orders of Katy

• Alternative was an enhanced service panel but this was rejected

• Convert workflow model to SCUFL2

• Use platform for workflow runs

• Including run management (currently in 2.x this is in the UI !)

• Plugin system

• Potentially complex

• Updated Monitoring & Provenance systems

Taverna 3 Hackathon• October 2012

• Alan (4 weeks)

• David (4 weeks)

• Stian (3 weeks)

• Donal (2 weeks)

• Rob (1 or 2 weeks)

• Aim to produce a runnable Taverna 3 workbench