carmen and spike detection and sorting
DESCRIPTION
CARMEN and spike detection and sorting . Leslie S. Smith University of Stirling, Scotland, UK http:// www.carmen.org.uk. Contents. CARMEN architecture and project Neural Data Format (NDF) Workflows (?) Project status Where we are now Some reflections. CARMEN ‘Cloud’ (CAIRN). - PowerPoint PPT PresentationTRANSCRIPT
CARMEN and spike detection and sorting
Leslie S. SmithUniversity of Stirling, Scotland, UK
http://www.carmen.org.uk
INCF June 17 2012 2
Contents• CARMEN architecture and project• Neural Data Format (NDF)• Workflows (?)• Project status
– Where we are now• Some reflections
CARMEN ‘Cloud’ (CAIRN)
Raw Signal Data Search & Visualisation
Data
Metadata
Compute Cluster on which Services
are DynamicallyDeployed
Web
Port
al
Rich Clients
Secu
ri ty
Workflow Enactment
Engine
RegistryService
Repository
Enactment of scientific analysis processes
Security Policies Controlling Access to Data & Code Search for Data &
Analysis Code
Structured Metadata Store Enabling Search & Annotation
AnalysisCode Store
Raw & Derived Data Store
INCF June 17 2012 4
CARMEN project status
• Initially a 4 year UK e-Science project– From September 2006-March 2011
• Extended with BBSRC tools and techniques grant– To Sept 2014
• Major work in last 2 years has been– Improving User Interface– NDF implementation (working)– Workflow Implementation (nearly there!)
INCF June 17 2012 5
CARMENCARMEN and spikes
Note that CARMEN also has other services, including higher level services, etc.
6
Data, services and workflows• CARMEN supports
– Data and metadata– Services: which process data, and– workflows (almost): concatenations of services.
• Initially: – We allowed more or less any data format– Services which processed a data format and produced a different
data format• …but to develop workflows (and to enable interoperability
between services)– We now strongly recommend using our Neural Data Format (NDF)
INCF June 17 2012
INCF June 17 2012 7
Neural Data Format (NDF)
• An NDF dataset consists of a configuration file in XML format which contains metadata and references to the associated host data files.
• A special XML data element, History, is included within the header file for recording data processing history. This element contains the full history (recording chain) of previous processing
• The NDF API has been implemented as a C library. The NDF API translates the XML tree/nodes to C style data structures and insulates the data structures within the binary data file from the clients.
INCF June 17 2012 8
NDF: Supported datatypes
9
NDF XML file<?xml version="1.0" encoding="UTF-8" standalone="no" ?><ndtfDataCfg xmlns="http://www.carmen.org.uk" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.carmen.org.uk ndtfDataCfg.xsd"> <Version>1.0.1</Version> <NdtfDataID>897A9272-4E6F-4F32-8A63-89C699F99120</NdtfDataID> <GeneralInfo> <Description>NDF Spike Detector Service (COB) Version 2 - SNN</Description> <Laboratory>Carmen VLE</Laboratory> <CreateDate>2012-06-13</CreateDate> <CreateTime>16:29:03</CreateTime> <RecordID>N/A</RecordID> </GeneralInfo> <DataSet>…
INCF June 17 2012
INCF June 17 2012 10
NDF XML file cont’d <History> <Processor> <ProcessingDateTime StartDateTime="2012-06-01T16:16:08"/> <CommandLine>mat2ndf (m0192_all.mat,m0192_all.ndf)</CommandLine> </Processor> <Processor> <ProcessingDateTime StartDateTime="2012-06-13T17:05:58"/> <CommandLine>Spike detector COB NDF m0192_all.ndf, 256, 0.002, 15, 30, no</CommandLine> <ProcessingSettings>Spike Detector (COB) for NDF data</ProcessingSettings> </Processor> </History> </ndtfDataCfg>
INCF June 17 2012 11
NDF based services• Filtering services:
– HPF, LPF, BPF• Spike detectors:
– single or multiple channel signal – Simple thresholding, positive/negative/both sided, NEO (Teager energy
operator), Cepstrum of Bispectrum• Spike sorters
– Kmeans– Waveclus (superparamagnetic clustering)
• We can add new spike detectors and new spike sorters reasonably easily. – Wrapping services
• The User Interface allows specific channels and sections of the dataset to be selected
INCF June 17 2012 12
Workflows
• Currently at alpha stage testing: – Can create workflows (graphically), generate scripts,
store them, apply security and sharing appropriately: execution of workflows is almost ready.
• Workflows will be generable either graphically or using a scripting language.
INCF June 17 2012 13
Workflow graphical interface
14
Where are we now? On the cluster
• Can run single services:– But joining them together requires user intervention– (NDF services do read each other’s data correctly)– New NDF services can be “wrapped”
• Have run workshops on this
• Can turn a variety of formats into NDF– Mcd, nev, nex, plx, map, smr, abf, abf2
• Spike detectors of three sorts– Can process multi-electrode data in one service
• Spike sorters of two sorts (Kmeans and waveclus)• But no workflows yet (promised soon!)
– … also not really enough public datasets
INCF June 17 2012
INCF June 17 2012 15
Where are we now? Local systems
• NDF toolbox is available for Matlab (downloadable). – Runs on recent versions of Matlab: PC, Mac, Linux.
• Services which run on the cluster are/will shortly be available to run locally under Matlab– Not really the intent of the project, but does enable
service running and testing (and debugging!) to be carried out locally
• Environment is essentially the same as on the cluster.• Local workflows enabled through writing XML files, and simple (-
ish) scripts.– Can test the wrapping of scripts locally
INCF June 17 2012 16
CARMEN and validation
• Validation of services• Testing services on
multiple datasets• Testing multiple services
on datasets– Locally– On the Portal.
INCF June 17 2012 17
Why is this so difficult? Why has it taken so long?What lessons can we learn?
• Initially allowed user to write services for their own data types– Ties services to specific types: not easily shareable
• NDF proved complex to implement– Generality, multiple language support
• User Interface for portal was difficult– Wanted to support non-technical users
• Existing software proved difficult to use– Software developed for R&D systems proved not to be robust: expecting
it to be was overly optimistic• Insufficient development staff
– Underestimated programming/development requirements– Supporting early development projects (“low hanging fruit”) took a lot of
time.
INCF June 17 2012 18
Carmen consortium