1

Automatic Processing Pipelines with XNAT and REDCap

Vanderbilt University

2013-08-20

Benjamin Yvernault,Bennett Landman, Brian Boyd,

2

RequirementsAt this point, you should know how to :

– Create project / sharing data– Browse data and view image previews – Search for projects, subjects, and data– Download imaging data, processed, and stored meta-data– Delete scans / sessions / projects– Use the Pre-Archive– Upload scans DICOMs – Quality Assurance

3

After this short course, you will be able to:

• Create a project on REDCap • Set up the design of your REDCap project• Request API access for your project on REDCap• What is a spider:

- source code - Which spiders already exist?- Run spiders from your computer

• Run a spider on your local machine• ACCRE: run spiders (status / failed / ….)

4

Setting REDCap project for XNAT processing

5

REDCap: Setup the main instrument

Variable Name: project_xnat -> type: Text BoxVariable Name: subject_xnat -> type: Text BoxVariable Name: experiment_xnat -> type: Text BoxVariable Name: scan_xnat -> type: Text BoxVariable Name: scan_type_xnat -> type: Text BoxVariable Name: scan_sd_type -> type: Text BoxVariable Name: script_name -> type: Text BoxVariable Name: script_version -> type: Text BoxVariable Name: pdf_report -> type: File Upload

6

REDCap: Upload main instrument

Variable Name: project_xnat -> type: Text BoxVariable Name: subject_xnat -> type: Text BoxVariable Name: experiment_xnat -> type: Text BoxVariable Name: scan_xnat -> type: Text BoxVariable Name: scan_type_xnat -> type: Text BoxVariable Name: scan_sd_type -> type: Text BoxVariable Name: script_name -> type: Text BoxVariable Name: script_version -> type: Text BoxVariable Name: pdf_report -> type: File Upload

Take time to create all this fields each time you have a new project.

Upload the main instrument dictionary as a first step. File location: http://xnat.vanderbilt.edu -> Everything about spiders -> How to upload to REDCap -> link with the csv file.

7

REDCap: Download Shared Libraries

8

REDCap: API Key To send data from XNAT to REDCap, you need to have

an API key for your project on REDCap

9

REDCap Project Online Designer

10

Spiders … Script executing a process on XNAT

Download images / data from XNATExecute a process on themCopy the data to a directory

A specific spider to upload results on XNAT All the spiders on http://xnat.vanderbilt.edu Source code in the project Masimatlab on NITRC Three kind of Spiders :

Spider that creates results for a ScanSpider that corresponds to an assessor Infrastructure spiders

11

Scan SpidersConverter Spider :

Spider_dcm2nii_all.py (using dcm2nii)Spider_dcm2nii_all_v2.py (using r2agui)

Changing attributes:Spider_set_scan_type.pySpider_Series_Description.py

Extracting/creating files:Spider_physlog.pySpider_Make_Preview_from_nifti.py

12

Process SpidersThis spiders can work on a scan level (one assessor per

scan) or an experiment level (one assessor per session)Spider that generates an assessor:

Spider_fMRQA.pySpider_VBMQA.pySpider_dtiQA_v2.pySpider_labelFusion.pySpider_intra_sess_reg.pySpider_nonrigi_reg_to_ATLAS.pySpider_FreesurferReconAllWithQC.pySpider_ON_segmentation.py

13

Infrastructure SpiderSpider_report_for_projects.py to create a report on

projects (same than Xnatreport)

Spider_send_data_to_redcap.py to send data from complet processes on XNAT to REDCap

Spider_Process_Upload to upload the processes that finished. Any process Spiders will copy the results needed to a specific directory. You should run it when you need to upload processes.

14

Spider flow

Spiders

Python script

Any language script: - Matlab/SPM - Perl - shell script

Folder

UPLOAD_SPIDER_DIR

Download data

Run process

Spider_process_upload.py / Xnatprocessupload

15

How to run a spider?

Spiders can run on your local machine if you have the requirements.E.G: Most of the spiders use xvfb-runpython Spider_fMRIQA.py -h to see the optionsYour .bashrc or .profile set up with your XNAT logsPython packages install

Specific python package to run spiders on ACCRE develop by our lab.

16

ACCRE and Spiders Advanced Computing Center for Research & Education ACCRE is a Cluster for Vanderbilt University Python script running every night on the cluster Submitting one job per assessor Status on the assessor related to the job status:

NEED_INPUTS-> the inputs files are not present on XNAT NEED_TO_RUN-> ready to be submitted on ACCRE JOB_RUNNING -> job submitted and running on ACCRE JOB_FAILED -> job stop with an error READY_TO_UPLOAD-> job ended and copy the results to the Upload directory UPLOADING-> Spider_Process_Upload running and uploading results READY_TO_COMPLETE-> a script will check and set the walltime and memory

used by the job on ACCRE. COMPLETE-> jobs done (processes finished and uploaded) with results on XNAT

17

ACCRE and CCI package

CCIPackage

Crontab job running every night a python script

Folder

UPLOAD_SPIDER_DIR

Get assessors that need to run

Run jobs for each assessors (Spider flow on ACCRE)

Spider_process_upload.py / Xnatprocessupload

18

Status sync between XNAT and REDCap

19

Interactive Assignment

• Add to your .bashrc the line (or .profile on Mac OS): export UPLOAD_SPIDER_DIR=path

• Create a project on REDCap• Upload the dictionary from http://xnat.vanderbilt.edu

– Everything about Spiders -> How to upload on REDCap

• Download the shared libraries for fMRIQA / VBMQA / dtiQA• Run the Spider VBMQA on one T1 in the project you created in

the last courses (it’s going to take 10mins)• Look at the results and upload them using Xnatprocessupload• Delete the project