![Page 1: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/1.jpg)
WACRunning CAW with AWS BatchHopefully my last talk about CAW ever...
Maxime Garcia @gau @MaxUlysse https://maxulysse.github.io/2018-01-31
SciLifeLab NGI / BarnTumörBanken
![Page 2: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/2.jpg)
Definition
1/8
![Page 3: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/3.jpg)
What is CAW?
WAC http://opensource.scilifelab.se/projects/caw/
• Nextflow pipeline
• Developed at NGI• In collaboration with NBIS• Support of The Swedish Pediatric Tumor Biobank
2/8
![Page 4: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/4.jpg)
What is CAW?
WAC http://opensource.scilifelab.se/projects/caw/
• Nextflow pipeline• Developed at NGI
• In collaboration with NBIS• Support of The Swedish Pediatric Tumor Biobank
2/8
![Page 5: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/5.jpg)
What is CAW?
WAC http://opensource.scilifelab.se/projects/caw/
• Nextflow pipeline• Developed at NGI• In collaboration with NBIS
• Support of The Swedish Pediatric Tumor Biobank
2/8
![Page 6: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/6.jpg)
What is CAW?
WAC http://opensource.scilifelab.se/projects/caw/
• Nextflow pipeline• Developed at NGI• In collaboration with NBIS• Support of The Swedish Pediatric Tumor Biobank
2/8
![Page 7: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/7.jpg)
What does CAW do?
WAC
• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files
• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller
• Structural variants
• Manta
• Heterogeneity, ploidy and CNVs
• ASCAT
• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 8: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/8.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)
• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files
• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller
• Structural variants
• Manta
• Heterogeneity, ploidy and CNVs
• ASCAT
• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 9: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/9.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38
• Based on GATK best practices for processing FASTQ files
• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller
• Structural variants
• Manta
• Heterogeneity, ploidy and CNVs
• ASCAT
• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 10: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/10.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files
• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller
• Structural variants
• Manta
• Heterogeneity, ploidy and CNVs
• ASCAT
• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 11: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/11.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller
• Structural variants
• Manta
• Heterogeneity, ploidy and CNVs
• ASCAT
• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 12: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/12.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller
• Structural variants
• Manta• Heterogeneity, ploidy and CNVs
• ASCAT
• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 13: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/13.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller• Structural variants
• Manta• Heterogeneity, ploidy and CNVs
• ASCAT• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 14: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/14.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller• Structural variants
• Manta
• Heterogeneity, ploidy and CNVs
• ASCAT
• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 15: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/15.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller• Structural variants
• Manta• Heterogeneity, ploidy and CNVs
• ASCAT• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 16: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/16.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller• Structural variants
• Manta• Heterogeneity, ploidy and CNVs
• ASCAT
• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 17: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/17.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller• Structural variants
• Manta• Heterogeneity, ploidy and CNVs
• ASCAT• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 18: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/18.jpg)
What does CAW do?
WAC• WGS analysis (Tumor/Normal pair or Germline)• Handles both GRCh37 and GRCh38• Based on GATK best practices for processing FASTQ files• SNPs, SNVs and indels
• MuTect1, MuTect2, Strelka, and GATK HaplotyeCaller• Structural variants
• Manta• Heterogeneity, ploidy and CNVs
• ASCAT• Containers (portable, reproducible)
• Docker or Singularity
3/8
![Page 19: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/19.jpg)
Where to use CAW?
• Any POSIX compatible system
• Rackham• Bianca• Irma
• AWS Batch
4/8
![Page 20: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/20.jpg)
Where to use CAW?
• Any POSIX compatible system
• Rackham
• Bianca• Irma
• AWS Batch
4/8
![Page 21: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/21.jpg)
Where to use CAW?
• Any POSIX compatible system
• Rackham• Bianca
• Irma
• AWS Batch
4/8
![Page 22: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/22.jpg)
Where to use CAW?
• Any POSIX compatible system
• Rackham• Bianca• Irma
• AWS Batch
4/8
![Page 23: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/23.jpg)
Where to use CAW?
• Any POSIX compatible system
• Rackham• Bianca• Irma
• AWS Batch
4/8
![Page 24: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/24.jpg)
Where to use CAW?
• Any POSIX compatible system• Rackham• Bianca• Irma
• AWS Batch
4/8
![Page 25: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/25.jpg)
Where to use CAW?
• Any POSIX compatible system• Rackham• Bianca• Irma• AWS Batch
4/8
![Page 26: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/26.jpg)
CAW with AWS Batch
• A single command line
nextflow run main.nf -profile awsbatch -w s3://caw-test-results/work \--genome smallGRCh37 --sample s3://caw-test-data/tsv/tiny-s3.tsv \--outDir s3://caw-test-results/Results
https://maxulysse.github.io/2017/11/16/Running-CAW-with-AWS-Batch/
5/8
![Page 27: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/27.jpg)
CAW with AWS Batch
• A single command line
nextflow run main.nf -profile awsbatch -w s3://caw-test-results/work \--genome smallGRCh37 --sample s3://caw-test-data/tsv/tiny-s3.tsv \--outDir s3://caw-test-results/Results
https://maxulysse.github.io/2017/11/16/Running-CAW-with-AWS-Batch/
5/8
![Page 28: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/28.jpg)
CAW with AWS Batch
• A single command line
nextflow run main.nf -profile awsbatch -w s3://caw-test-results/work \--genome smallGRCh37 --sample s3://caw-test-data/tsv/tiny-s3.tsv \--outDir s3://caw-test-results/Results
https://maxulysse.github.io/2017/11/16/Running-CAW-with-AWS-Batch/
5/8
![Page 29: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/29.jpg)
Going further
• Run a full size test sample
• Gather reports• Get a pricing• Wait for Amazon to finally come to Stockholm
6/8
![Page 30: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/30.jpg)
Going further
• Run a full size test sample• Gather reports
• Get a pricing• Wait for Amazon to finally come to Stockholm
6/8
![Page 31: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/31.jpg)
Going further
• Run a full size test sample• Gather reports• Get a pricing
• Wait for Amazon to finally come to Stockholm
6/8
![Page 32: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/32.jpg)
Going further
• Run a full size test sample• Gather reports• Get a pricing• Wait for Amazon to finally come to Stockholm
6/8
![Page 33: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/33.jpg)
The List of People Involved
Sebastian DiLorenzo Markus MayrhoferJesper Eisfeldt Monica NistèrPhil Ewels Björn NystedtMaxime Garcia Pall OlasonSzilveszter Juhos Markus RingnérMax Käller Pelin SahlénMalin Larsson Johanna SandgrenMarcel Martin Teresita Díaz De Ståhl
7/8
![Page 34: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/34.jpg)
Where to find us?
• We are on the SciLifeLab Slack
#cancer-pipeline
• We have a gitter channel https://gitter.im/SciLifeLab/CAW
• Our code is hosted on Github
https://github.com/SciLifeLab/CAW
8/8
![Page 35: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/35.jpg)
Where to find us?
• We are on the SciLifeLab Slack
#cancer-pipeline
• We have a gitter channel https://gitter.im/SciLifeLab/CAW
• Our code is hosted on Github
https://github.com/SciLifeLab/CAW
8/8
![Page 36: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/36.jpg)
Where to find us?
• We are on the SciLifeLab Slack
#cancer-pipeline
• We have a gitter channel https://gitter.im/SciLifeLab/CAW
• Our code is hosted on Github
https://github.com/SciLifeLab/CAW
8/8
![Page 37: Running CAW with AWS Batch...2018/01/31 · • Docker or Singularity 3/8 What does CAW do? CAW • WGS analysis (Tumor/Normal pair or Germline) • Handles both GRCh37 and GRCh38](https://reader034.vdocuments.us/reader034/viewer/2022042802/5f428376a95638427c771b82/html5/thumbnails/37.jpg)
Any questions?