RESEARCH Open Access

Sequencing of animal viruses: quality dataassurance for NGS bioinformaticsGianpiero Zamperin1, Pierrick Lucas2,3, Irene Cano4, David Ryder4, Miriam Abbadi1, David Stone4, Argelia Cuenca5,Estelle Vigouroux3,6, Yannick Blanchard2,3* and Valentina Panzarin1*

Abstract

Background: Next generation sequencing (NGS) is becoming widely used among diagnostics and researchlaboratories, and nowadays it is applied to a variety of disciplines, including veterinary virology. The NGS workflowcomprises several steps, namely sample processing, library preparation, sequencing and primary/secondary/tertiarybioinformatics (BI) analyses. The latter is constituted by a complex process extremely difficult to standardize, due tothe variety of tools and metrics available. Thus, it is of the utmost importance to assess the comparability of resultsobtained through different methods and in different laboratories. To achieve this goal, we have organized aproficiency test focused on the bioinformatics components for the generation of complete genome sequences ofsalmonid rhabdoviruses.

Methods: Three partners, that performed virus sequencing using different commercial library preparation kits andNGS platforms, gathered together and shared with each other 75 raw datasets which were analyzed separately bythe participants to produce a consensus sequence according to their own bioinformatics pipeline. Results werethen compared to highlight discrepancies, and a subset of inconsistencies were investigated more in detail.

Results: In total, we observed 526 discrepancies, of which 39.5% were located at genome termini, 14.1% atintergenic regions and 46.4% at coding regions. Among these, 10 SNPs and 99 indels caused changes in theprotein products. Overall reproducibility was 99.94%. Based on the analysis of a subset of inconsistenciesinvestigated more in-depth, manual curation appeared the most critical step affecting sequence comparability,suggesting that the harmonization of this phase is crucial to obtain comparable results. The analysis of a calibratorsample allowed assessing BI accuracy, being 99.983%.

Conclusions: We demonstrated the applicability and the usefulness of BI proficiency testing to assure the quality ofNGS data, and recommend a wider implementation of such exercises to guarantee sequence data uniformityamong different virology laboratories.

Keywords: NGS, Bioinformatics, Proficiency testing, Virology

BackgroundIn veterinary medicine, diagnosis, monitoring and preven-tion of infectious diseases can no longer neglect to performan accurate genetic characterization of their causativeagents [1]. In fact, the number of molecular markers of

pathogens with known diagnostic or prognostic value hasrapidly increased, allowing a more complete and in-formative analysis [2–4]. On the other hand, diseasesglobalization has required diagnostic laboratories to facethe challenge of staying at the forefront of research andtechnology, in order to increase their preparedness indetecting emerging pathogens and new genetic markersresponsible for different virulence phenotypes.In this scenario, the advent of next generation sequen-

cing (NGS) technologies has offered an unprecedentedopportunity to generate sequence data in a high-throughput fashion, with or without prior knowledge of

© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

* Correspondence: yannick.blanchard@anses.fr; vpanzarin@izsvenezie.it2French Agency for Food, Environmental and Occupational Health & Safety(ANSES), Ploufragan-Plouzané-Niort Laboratory, Viral Genetics and BiosecurityUnit, 22440 Ploufragan, France1Department of Comparative Biomedical Sciences, Istituto ZooprofilatticoSperimentale delle Venezie (IZSVe), viale dell’Università 10, 35120 Legnaro(PD), ItalyFull list of author information is available at the end of the article

Zamperin et al. Virology Journal (2019) 16:140 https://doi.org/10.1186/s12985-019-1223-8

the pathogen involved. For these reasons, NGS has be-come the elected tool for many laboratories performingdiagnostic assays, and is being extensively applied toveterinary medicine as well [4, 5]. However, despite theundisputed advantages of this technology, NGS assaysinvolve multifaceted workflows (sample preparation,tissue enrichment, nucleic acid isolation, library prepa-ration, quantitation and pooling, sequencing via uniquechemical processes, data analysis) and therefore is poten-tially prone to the introduction of errors throughout thewhole analytical process. In particular, the considerablevolume of raw data produced requires complex bioinfor-matics (BI) analyses that, in turn, necessitate a greatamount of computational power in order to extract bio-logical significance from the sequenced samples. For thisreason, the BI analysis appears particularly vulnerableand requires appropriate quality checks to ensure thereliability of the results [5]. The steps implicated in thisphase can be categorized into three stages: primary,secondary and tertiary analyses [6, 7]. Typically, primaryanalysis is performed directly in the sequencing platformand involves the conversion of raw signals into base callsand sequence reads. The conversion is carried out bysoftware integrated within the instrument and, if neces-sary, it is followed by demultiplexing of pooled data, i.e.assignment of reads to the related sample. The output ofthe machine sequencer is the result of the primary ana-lysis and it is referred to as raw data. Secondary analysisgreatly varies in respect to sample type and researchaim, and generally comprises a) filtering steps to removepoor data; b) assembly of high quality reads, either by areference-based or de novo approach; c) variant detec-tion. Tertiary analysis strongly depends on the field ofapplication and consists of human-driven interpretationof sequence data. All these computational steps use fil-ters and quality control (QC) thresholds to convert con-tinuous measurements into discrete results. Notably, thechoice of QC metrics can greatly affect the outcome ofthe BI pipeline, even among laboratories employing thesame tools within the same workflow. Additionally, ma-nual curation and interpretation of data at all stages isoften implemented, particularly during tertiary analysis,adding another layer of variability. Such diversity andflexibility, both at the computational and interpretationlevel of the BI analysis, makes it challenging to defineuniversal metrics to assure the quality of sequence data.However, it is of utmost importance to evaluate theperformance of post-sequencing analyses as part of thequality management of every laboratory using NGS.Although exist software programs that allow to per-

form intra-laboratory quality check for basic issues alongdifferent stages of the BI analysis, proficiency testing(PT) remains the most appropriate instrument to assessthe comparability of sequence data among different

laboratories using diverse analytical methodologies.While PTs are common tools for the evaluation of thewet-bench part, they are still rarely employed for the insilico steps of the NGS analysis, most likely because ofthe high variability of this phase, the availability of newsoftware and, consequently, the difficulty in establishingstandard guidelines [8–10]. In the present study, weconducted for the first time a PT on NGS applied to vet-erinary virology based on real data, aimed at evaluatingthe BI pipeline for consensus sequence generation re-gardless of the techniques adopted for library productionand sequencing. This work was carried out within theframework of the Novimark project (Anihwa ERA-Net)(https://www.anihwa-submission-era.net/novimark.html;https://www.izsvenezie.com/novimark-novirhabdo-viruses-trouts-salmons/), that aims at identifying viru-lence markers of two Novirhabdoviruses, the viralhaemorrhagic septicaemia virus (VHSV) and the infec-tious haematopoietic necrosis virus (IHNV), by integrat-ing phenotypic and genetic data related to field strainsthat originate from different laboratories in Europe.These viruses are of particular concern because they areresponsible for two OIE notifiable diseases that severelyaffect European trout industry, namely VHS and IHN.Fundamental to the Novimark scope was the reliabilityof viral consensus sequences, as different institutionswithin the consortium produced genetic data with theNGS technologies and BI pipelines available at their fa-cilities. To reach this goal, sequenced reads generated bythree laboratories were gathered together to constitute aunique set of 75 raw data, which was then shared amongthe participants for BI analysis. The dataset encom-passed either VHSV or IHNV field strains as well as onerecombinant VHSV. Both viral species have the samegenome architecture, constituted by a negative-sense,single-stranded RNA molecule consisting of six genes inthe order 3′-N-P-M-G-NV-L-5′. They encode six struc-tural and non-structural proteins, namely the nucleocap-sid protein (N), the phosphoprotein (P), the matrixprotein (M), the glycoprotein (G), the non-virion protein(NV) and the polymerase protein (L) [11].The participants were asked to produce a consensus

sequence for each strain according to their own BImethodology, and complete genomes were thencompared to highlight any discrepancy in terms oftype (Single Nucleotide Polymorphisms - SNPs anddeletions/insertions - indels) and localization (codingregions, intergenic regions, genome termini). For arandomly selected subset of discrepancies, data werefurther explored in order to identify analytical stepsof the BI pipeline responsible for inconsistent resultsamong laboratories. In this study, we have demon-strated the feasibility of in silico PT applied to vete-rinary virology and proposed a general flowchart to

Zamperin et al. Virology Journal (2019) 16:140 Page 2 of 13

assess the similarity of outputs from different labora-tories using diverse BI pipelines.

MethodsStudy designSeventy-three VHSV (n = 55) and IHNV (n = 18) viralstrains collected by the Novimark consortium were usedfor the exercise [Additional file 1] and processed bythree participant laboratories, later referred as Lab 1,Lab 2 and Lab 3, according to the protocols and sequen-cer machines available at their facilities. In detail, 36VHSV and 13 IHNV were sequenced with IlluminaMiSeq by Lab 2 and Lab 3, and 19 VHSV and 5 IHNVwere sequenced with Ion Proton™ Sequencer by Lab 1.Additionally, a calibrator specimen constituted by a re-combinant VHSV strain (see below) was included in theexercise and analyzed with both sequencing technologies(Fig. 1). All the viruses were subject to library prepa-ration and whole genome sequencing (WGS) accordingto the protocols reported below. In total, 75 unique rawdatasets, later referred as “raw data” or “sample(s)”, weregenerated by Lab 1, Lab 2 and Lab 3 according to themethodologies implemented at their facilities, andshared in FASTQ format, either as a single file or as twopaired files, via a secure File Transfer Protocol (FTP)site. For each sample, participants applied their own BIpipeline and produced a single consensus sequence forcomparison.

Specimens processing and sequencingLab 1Viral strains available at Lab 1 repository [Additionalfile 1], originally lyophilized or frozen at − 80 °C, wereinoculated at a multiplicity of infection (MOI) of 2–10onto 25 cm2 flasks seeded with blue gill fry (BF-2) or

epithelioma papulosum cyprini (EPC) cells [12, 13]grown in Eagle’s minimal essential medium adjusted atpH 7.6 ± 0.2 with 0.19M Tris-HCl buffer and supple-mented with 10% fetal bovine serum (Eurobio), 100 U/mL penicillin and 100 μg/mL streptomycin (Pan Bio-tech). After 1-h adsorption at 14 °C ± 1, inocula wereremoved and replaced with fresh medium. Cell cultureswere then incubated at 14 °C and checked regularly forthe development of cytopathic effect (CPE). Upon com-pletion of CPE, flasks were frozen and thawed, and viralsuspensions were collected and clarified at 2000×g for15 min at 5 °C ± 3 °C. Supernatants were then aliquotedand stored at − 80 °C until use.For each specimen, total RNA was extracted from 1ml

of supernatant using TRIzol™ LS Reagent (Life Tech-nologies) following the manufacturer’s instructions.Double stranded complementary DNA (ds-cDNA) li-braries were prepared with the Ion Total RNA-Seq KitV2 (Life Technologies). The protocol recommended bythe supplier was slightly modified and is available fromthe authors upon request. Finally, cDNA libraries werequality-checked with Agilent 2100 Bioanalyzer (AgilentHigh Sensitivity DNA kit, Agilent Technologies), quanti-fied by qPCR (Ion Library TaqMan™ Quantitation Kit,ThermoFisher Scientific) and finally sequenced on theIon Proton™ Sequencer using an Ion PI™ Chip Kit v2(Life Technologies).

Lab 2Viral stocks provided by the European Union Refer-ence Laboratory for Fish and Crustacean Diseases(Denmark) and processed by Lab 2 [Additional file1], were freshly propagated at 15 °C on exponentiallygrowing EPC cells [13] cultivated in 75 cm2 flaskswith L-15 cell medium supplemented with 1 mM L-

Fig. 1 Samples description. Outer and inner rings show samples distribution with respect to viral strain (VHSV, IHNV, calibrator) and tosequencing technology (Illumina MiSeq, Ion Proton™), respectively

Zamperin et al. Virology Journal (2019) 16:140 Page 3 of 13

glutamine, 2% fetal bovine serum, 100 U/ml penicil-lin and 200 μg/ml streptomycin (Gibco). Once fullCPE was obtained, culture supernatants were col-lected, and cell debris removed by centrifugation at2500×g for 15 min at 4 °C. Viruses were then con-centrated by ultracentrifugation at 100000×g for 1 hat 4 °C, and pellets re-suspended in 200 μl of cellculture medium.Viral RNA was extracted using the EZ1 Virus Mini Kit

v2.0 and the EZ1 Advanced extraction robot (Qiagen),and eluted in 60 μl of elution buffer. Approximately 100ng of RNA were reverse transcribed at 37 °C for 1 h in a20 μl reaction containing 1mM dNTPs, 500 ng of randomprimers and 200 unitsM-MLV Reverse Transcriptase(Promega). Double-stranded cDNA was then generatedwith random primers using Sequenase V2.0 DNA Poly-merase (Affymetrix), purified with the QIAquick PCRPurification Kit (Qiagen) and quantitated using Quantus™Fluorometer (Promega). Libraries were prepared with theNextera XT DNA Library Preparation Kit (Illumina),checked for quality and size with Agilent 2100 Bioanalyzer(Agilent High Sensitivity DNA kit, Agilent Technologies)and sequenced with Miseq v2 Reagent Kit (150PE)(Illumina) using Illumina MiSeq platform.

Lab 3Three ml of each virus originating from Lab 3 repository[Additional file 1] were inoculated onto 24-h EPC cellsseeded in 72 cm2 flasks (Falcon®). After 1-h adsorption at15 °C under gentle shacking, 15ml of MEM Eagle (Sigma-Aldrich) supplemented with 10% foetal calf serum (FCS)(Hyclone), 1% L-glutamine 200mM (Sigma-Aldrich) and1% antibiotic antimycotic solution 100X (Sigma-Aldrich)were added, and viruses were propagated at 15 °C untilcompletion of CPE. Approximately 20ml of viral suspen-sion were collected from each flask and clarified at 4 °Cfor 10min at 2800×g. Subsequently, viruses were concen-trated by ultracentrifugation at 90000×g for 1 h using aBeckman Coulter Optima L-100 K, and then re-suspendedin 500 μl of MEM Eagle (Sigma-Aldrich).Viral RNA was isolated from 140 μl of concentrated

virus using the QIAamp Viral RNA Mini kit (Qiagen) andthen subject to retrotranscription with the SuperScript IIIReverse Transcriptase (Invitrogen). Double-strandedcDNA was synthesized using NEBNext mRNA SecondStrand Synthesis Module (Euroclone), purified withMagSI-NGSPREP PLUS beads (MagnaMedics) and quan-tified with Qubit dsDNA HS assay kit (ThermoFisher Sci-entific). The cDNA library was prepared using IlluminaNextera XT DNA Sample Preparation kit (Illumina), andfragments were selected with MagSI-NGSPREP PLUSbeads (MagnaMedics). Library was checked for qualityand size with Agilent 2100 Bioanalyzer (Agilent High Sen-sitivity DNA kit, Agilent Technologies) and sequenced

with MiSeq v3 Reagent Kit (300PE) using Illumina MiSeqplatform.

Calibrator production and sequencingThe recombinant VHSV strain r23/75 kindly providedby the Institut National de la Recherche Agronomique(INRA) was synthetized via reverse genetics by transfect-ing EPC cells with an expression plasmid containing theVHSV full-length cDNA [14]. Being a recombinant viruswith a predetermined genome sequence, strain r23/75acted as calibrator for the exercise and was subject toNGS with both Ion Proton™ and Illumina MiSeq accord-ing to the procedures available at Lab 1 and Lab 3,respectively. To further verify its sequence, the vectorencoding r23/75 genome (p23/75) and used for EPCtransfection was also Sanger sequenced by Lab 3. Briefly,the plasmid was propagated in XL 10-Gold ultracompe-tent cells (Agilent Technologies) and isolated using theEndoFree Plasmid Maxi Kit (Qiagen). The purified vec-tor was then subject to sequencing using primers listedin Additional file 2 and the BrilliantDye™ TerminatorCycle Sequencing Kit (Nimagen) according to the manu-facturer’s instructions. Sequencing reactions were thenpurified with CENTRI-SEP 96 Well Plates (PrincetonSeparations) and sequenced in a 16-capillary ABI PRISM3130xl Genetic Analyzer (Applied Biosystems). Sequencedata were assembled and edited using the SeqScape soft-ware v2.5 (Applied Biosystems) and the final consensuswas compared with the genome sequences of r23/75 ob-tained with NGS. The VHSV complete genome of p23/75 is publicly available under the GenBank acc. no.MK792283.

Bioinformatics analysisLab 1Reads were cleaned with Trimmomatic v0.36 [15](ILLUMINACLIP: oligos.fasta: 2:30:5:1: true; LEADING:22; TRAILING: 22 for Ion Proton™ sequencing and 28for Illumina MiSeq sequencing; MAXINFO: 40:0.2;MINLEN: 36). Bowtie2 v2.2.5 [16] was adopted to per-form a rapid alignment with down-sampled reads on thelocal NT database. VHSV or IHNV complete genomeswith the highest number of matching reads were used asa primary reference to align all cleaned reads with BWAv0.7.15 [17]. Based on the alignment, the coverage wasestimated using an in-house perl script, bam stat v1.0.13(http://bamstats.sourceforge.net/) and bam2fastx v1.0(https://github.com/PacificBiosciences/bam2fastx). Rawreads were down-sampled to fit an estimated globalcoverage of 80X fold and were de novo assembled withMira v4.0.2 [18] and, after being cleaned with Trimmo-matic, also with SPAdes v3.10.0 [19]. Subsequently, as-sembled contigs were aligned against the integrated NTdatabase using BLAST [20]. The best match (i.e. the

Zamperin et al. Virology Journal (2019) 16:140 Page 4 of 13

longest sequence corresponding to a complete genomeof VHSV or IHNV) was selected as reference for a BWAalignment. The generated SAM file was then used tocompute the mpileup using samtools v1.9 [21], and vari-ant calling was performed with bcftools v1.9 [21]. Calledvariants were then used to produce a consensus se-quence with vcfutils.pl v1.9 [21] and seqtk v1.2 (https://github.com/lh3/seqtk). Finally, assembled contigs andBWA alignments were visually inspected for ambiguousbases, indels, ORF integrity, genome integrity and, ifrequired, were manually adjusted.

Lab 2Unless otherwise specified, pre-processing of raw data, readsalignment, variant calling and alignment of the consensussequences against the reference genomes were done usingCLC Genomics Workbench v4.9 (https://www.qiagenbioin-formatics.com/). Sequences were trimmed based on qualityscores with a cumulative error probability of 0.03, andincluding a maximum of 3 ambiguous bases. Sequencesshorter than 40 bp were discarded. Sequencing adaptorswere trimmed with mismatch and insertion costs of 2 and 3,respectively, and with a minimum internal and end scores of10 and 3, respectively. Reads were aligned against the follow-ing reference sequences: VHSV 23–75 (GenBank acc. no.FN665788), VHSV MI03GL (GenBank acc. no. GQ385941),VHSV SE-SVA-14-3D (GenBank acc. no. AB839745) andIHNV (GenBank acc. no. X89213). Mismatch, insertion anddeletion costs were set at 2, 3 and 3, respectively. At leasthalf of the reads was required to match the reference ge-nomes with a nucleotide similarity ≥80%. For those readsmapping to multiple positions within the reference genome,the position in the final alignment was randomly assigned.The consensus sequence was inferred based on the align-ment with the highest coverage. In case of disagreementamong reads, those predominantly represented were usedfor base calling. Once the drafted consensus sequence wasattempted, reads were re-aligned using the same criteria asbefore. Potentially deleterious mutations were examined bya) aligning the ultimate consensus sequence against the ref-erence with the highest similarity; b) listing any mismatchwith a putative detrimental effect on viral phenotype; c) pre-dicting the reading frames using Prokka v1.12 [22]. The sup-port for such deleterious mutations was obtained throughexamination of the original alignment and with comparisonagainst alternative variant calling tools, such as Snippy [23].

Lab 3Reads quality was assessed using FastQC v0.11.2 (https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).Illumina MiSeq raw data were cleaned by removing: a)reads with more than 10% of undetermined (“N”) bases; b)reads with more than 100 bases with Q score below 10; c)duplicated paired-end reads. Ion Proton™ raw data were

cleaned by removing duplicated reads. Filtered reads wereclipped from Illumina adaptors Nextera XT or Ion Pro-ton™ adaptors with scythe v0.991 (https://github.com/vsbuffalo/scythe). Low quality ends were trimmed withsickle v1.33 (https://github.com/najoshi/sickle) over arange with minimum average quality of 25 for IlluminaMiSeq reads, and 15 for Ion Proton™ reads. Reads shorterthan 80 bases or unpaired after previous filters were dis-carded. Filtered reads were aligned against the integratedNT database (version 8 February 2017) with BLASTv2.6.0+ adopting default parameters and e-value <10e-50.Alignment hits matching against VHSV and IHNVcomplete genomes, with sequence similarity ≥95% andmatch length of the query ≥99%, were selected. For eachsample, sequences with the highest number of matchingreads were chosen as proper reference genomes. Readswere then re-aligned against their respective referencegenome selected as above using BWA v0.7.12 and stand-ard parameters. Before SNPs calling, alignments wereprocessed with SAMtools v0.1.19 for conversion intoBAM format and sorted by position. Subsequently, poten-tial errors of the alignment were corrected with Picard-tools v2.1.0 (http://broadinstitute.github.io/picard/) andGATK v3.5 [24–26], reads were re-aligned in the proxim-ity of indels and base quality was recalibrated. LoFreqv2.1.2 [27] was then run with the function “--call-indels”to produce a vcf file containing both SNPs and indels.Indels with a frequency lower than 50% and SNPs with afrequency lower than 25% were discarded. Indels withincoding regions, determining a reading frameshift, werealso removed. The consensus sequence was obtainedadopting the following criteria: a) for coverage insufficientfor reliable variant calling (<10X), the “N” base wasassigned; b) for coverage ≥10X and no SNP call, the refer-ence genome base was assigned; c) for coverage ≥10X andcalling of at least one SNP, the nucleotide representingthe observed bases was assigned adopting the IUPACcode. High quality reads were re-aligned against their con-sensus sequence with BWA. Finally, the alignment wasvisually inspected with tablet v1.14.10.21 [28] and used tomanually revise the consensus sequence.

Consensus sequences comparisonThe consensus sequences produced by Lab 1, Lab 2 andLab 3 were saved in fasta format and collected for subse-quent analysis via a secure FTP site. Complete genomesof VHSV and IHNV were aligned separately withMAFFT v7.294b [29] using standard parameters. Differ-ences between the consensus sequences produced by theparticipants related to the same viral strain were identi-fied with a perl script developed in-house and availableupon request. For the recombinant calibrator, consensussequences obtained with NGS were compared also withthe Sanger sequence (Genbank acc. no. MK792283).

Zamperin et al. Virology Journal (2019) 16:140 Page 5 of 13

Discrepancies were evaluated by taking into accounttheir genome localization, namely a) coding regions(CDS), consisting of viral genes translated into viral pro-teins; b) non-coding intergenic regions, consisting of un-translated sequences that intersperse viral codingregions; c) genome termini, consisting of untranslatedregions at the 3′ and 5′ ends of the viral genome. Dis-crepancies were further categorized as SNPs or indels. Asummary of all the inconsistencies observed was gener-ated and scrutinized. PT reproducibility was estimatedas the ratio between the number of consistent sites ob-served and the cumulative genome size of the 75 sam-ples, as previously proposed by Kozyreva et al. (2017)[30]. Accuracy of the BI pipelines was assessed by pair-wise comparison of the consensus sequences related tothe calibrator sample against its reference genomeobtained with Sanger. The accuracy value was estimatedas the ratio between observed identical bases and theexpected genome size [30].To deepen the reasons behind inconsistent results, an

additional explanatory analysis was performed on a subsetof 50 discrepancies randomly selected. We intentionallyavoided discrepancies located at genome termini becausethe variability in the three BI pipelines adopted did notallow a proper assessment of the reasons for variant calls.The analysis was performed in three phases. Firstly, foreach discrepancy, participants were asked to provide ajustification for their choice in nucleotide assignment. Sec-ondly, all the explanations provided were ascribed to oneor a combination of key steps of the BI pipeline namely, i)alignment; ii) variant calling; iii) manual curation; iv) denovo assembly. The three laboratories performed steps i),ii) and iii) in the same sequential order to produce thefinal consensus sequence. Step iv) was unique to the BIpipeline of Lab 1 and was used during manual inspectionof the initial consensus sequence from step iii). Finally, toidentify the source of the discrepancies, the explanationsof the three laboratories were compared with respect tothe order of key steps described within the BI pipeline.Among the three explanations provided for each of the 50sites, the last by order in the steps series i-iv was arbitrarilyassumed to be the source of the inconsistency amonglaboratories.

ResultsComparison of the BI pipelinesIn this study, a set of 75 raw data related to 73 fish Novir-habdoviruses and one calibrator (Fig. 1) was shared amongthree laboratories to produce complete genomes accord-ing to their own BI pipeline. A flowchart summarizing thedifferent BI pipelines is available as supporting informa-tion [Additional file 3]. Lab 1 used a de novo assembly toidentify the best available complete genome for areference-based alignment; the consensus sequence

produced was then inspected and manually edited usingthe initial de novo assembly. Lab 2 used a reference-basedassembly on fixed reference sequences to produce an un-refined consensus, utilized as a reference genome for asubsequent assembly. A combination of manual and auto-matic checks was then performed to ensure the absence ofdeleterious mutations (i.e. nonsense or indels causingreading frameshift). Lab 3 used a reference-based assem-bly performed on the most suitable reference sequencepreviously identified. A final step of manual curation wasthen performed to ensure that the final consensus ob-tained was truly representative of the NGS data and toverify the maintenance of the correct reading frame. Un-doubtedly, the most heterogeneous steps of the BI pipelineamong laboratories were the selection of the referencegenome for the generation of the consensus, and its man-ual curation. The substantial differences in the analyticalapproach of each participant reflect the great variety oftools and BI pipelines, which in turn impedes thecomparison of QC metrics among groups.

Discrepancies analysisConsensus sequences obtained from the analysis of 75 rawdata were collected, and genomes referring to the same viralstrain were compared to highlight any inconsistency. Intotal, our analysis revealed 526 discrepancies, with an aver-age of 7.0 differences per sample (d/s). A distribution of alldiscrepancy types occurred, also in respect to sequencingtechnology, is shown in Fig. 2. A more detailed breakdownof the discrepancies observed in each sample is provided inAdditional file 4, where the number of inconsistencies isshown as regards type, genome location, viral species andsequencing technology. The ratio between the total numberof discrepant sites and the cumulative genome size for allthe raw data was 0.06%. This value was used as a means toevaluate PT reproducibility, corresponding to 99.94%. Whenconsidering sequencing technology, our analysis revealedthat raw data produced with Illumina MiSeq showed onaverage 4.5 d/s (227 differences over 50 samples), in contrastto Ion Proton™ that yielded on average 12 d/s (299 differ-ences over 25 samples). Although raw data obtained withIllumina MiSeq appeared to have a smaller number of dis-crepancies, it is worth noting that different strains were se-quenced with diverse systems, thus a direct comparison ofthe sequencing technology cannot be considered conclusive.Overall, 39.5% discrepancies among consensus se-

quences related to the same strain were located at gen-ome termini, 14.1% within intergenic regions and 46.4%affected the CDS. At genome termini, 208 differenceswere reported, 59.6% of which were SNPs and 40.4%were indels. The majority of these discrepancies wereobserved for viruses sequenced with Illumina MiSeq(110 SNPs and 58 indels over 50 samples, meaning 2.2SNPs and 1.2 indels per sample) in respect to Ion

Zamperin et al. Virology Journal (2019) 16:140 Page 6 of 13

Proton™ (14 SNPs and 26 indels over 25 samples, corre-sponding to 0.6 SNPs and 1.1 indels per sample) (Fig. 2a). NGS sequencing of genome termini is challenging,both for the wet-bench part and the in silico analysis. In-deed, fragmentation of these portions during librarypreparation produces fragments with a biased length dis-tribution. Thus, coverage at genome termini shows a de-clining profile that might dramatically affect the analysisof these regions. Additionally, for reference-basedassembly, also the choice of the reference genome is crit-ical for 3′ and 5′ ends reconstruction. For these reasons,we decided to exclude genome termini from the analysesto avoid an excess of complexity in reporting our resultsand to focus our attention only on SNPs and indels lo-cated at coding and intergenic regions. In intergenic re-gions, 74 differences were reported, 33 for IlluminaMiSeq data and 41 for Ion Proton™ data. The majority ofthe discrepancies (66%) was represented by small indels,with a rate of 14/50 (meaning 14 indels for 50 samples)for Illumina MiSeq data, and 35/25 for Ion Proton™ data.Twenty-five differences were attributable to SNPs, 9 ofwhich involved degenerate base symbols. We performeda more detailed investigation of these latter sites by

explicating all possible alternatives for the ambiguouscharacters detected. Then, for each of the 9 sites invol-ving degenerate base symbols, we checked if at least onealternative was in common between the three labs. Wefound that, considering degenerations in such a way, allgroups identified the same nucleotide. Thus, the discrep-ancies at these 9 sites were due to different thresholdvalues used for handling degenerate bases rather thandifferences in the variant calling. At coding regions, thecomparison of complete genomes produced with Illu-mina MiSeq identified 26 SNPs and no indel (Fig. 2 a).The “N” base was assigned to 3/26 SNPs by at least oneparticipant, impeding the unambiguous identification ofthe nucleotide. Twenty SNPs produced a synonymouscodon which resulted in no amino acid change in the re-lated protein. Contrarily, 3 SNPs (corresponding to 1.3%of all the differences observed for samples sequencedwith Illumina MiSeq) determined a non-synonymousmutation in the protein product. The analysis of codingregions related to consensus sequences produced withIon Proton™ revealed a higher number of differences(n = 218) (Fig. 2 a), 99 of which were indels and 119were SNPs. All the indels produced frameshift mutations

Fig. 2 Discrepancies distribution. a: Histograms show the total number of discrepancies counted in respect to type (SNP, indel) and genomelocalization (CDS, non-coding intergenic regions, genome termini); histograms are further divided to display total distribution, and distribution foreach type of sequencing technology (Illumina MiSeq, Ion Proton™). b: Histograms show discrepancies from the point of view of every participantlab; inconsistent sites are counted in respect to representing the majority or being the odd one out

Zamperin et al. Virology Journal (2019) 16:140 Page 7 of 13

with the disruption of most of the protein products andthe appearance of multiple premature stop codons. Ofnote, 98 indels were present in the consensus sequencesproduced by Lab 2 (Fig. 2 b), most likely due to a tech-nical error of the BI pipeline (i.e. variant caller tool). Theremaining indel was a 2 bp-long deletion called by Lab 1on a region with very poor coverage, for which it wasvery difficult to assess the reliability of the sequenceddata. Five SNPs out of 119 involved an “N” base for atleast one participant. Seven SNPs, representing 2.4% ofall the discrepancies observed in samples sequenced withIon Proton™, resulted in a missense mutation while 107SNPs produced a synonymous codon. It is worth notingthat approximately 80% (85/107) of the synonymousSNPs are attributable to strain 565-DK-297-HEDEDAMthat showed a high number of degenerations within theconsensus sequence generated by Lab 3, possibly due toa mixed viral infection. Because this sample accounts fora great amount of discrepancies, we explored the reasonsbehind this evidence in more depth. A direct inspectionof raw data showed that the minority variants frequencyfor the 85 ambiguous positions ranged between 14.3 and46.3%. The more conservative approach used by Lab 3with respect to filtering pipelines, alignments and variantcalling might explain the higher number of degenera-tions present in the complete genome of strain 565-DK-297-HEDEDAM (Figs. 2 b and S2).Coding regions of the consensus sequences obtained

were investigated also for their ability to produce a fullyfunctional protein. For all the samples but one, Lab 1produced complete genomes fully consistent in respectto CDS length, start/stop codon positions and absenceof premature stop codons due to SNPs or indels. Theonly exception was represented by one sample with a 2bp indel due to very few reads covering the site, result-ing in at least one premature stop codon that disruptsthe reading frame. Lab 2 produced identical results whenanalyzing Illumina MiSeq data; in contrast, the majorityof consensus sequences generated from Ion Proton™ dataheavily suffered from the presence of indels disruptingthe reading frame and introducing multiple prematurestop codons (Fig. 2). Lab 3 produced complete genomesfully consistent for all the samples with respect to CDSlength, start/stop codon positions and absence of prema-ture stop codons due to SNPs or indels, independentlyfrom the sequencing technology.To better compare the performances of each BI pipe-

line regardless of previous analytical steps, and assesstheir accuracy, we sequenced a calibrator specimen con-stituted by a recombinant VHSV using both the IlluminaMiSeq and Ion Proton™ sequencing platforms. Overall,the six genomes produced by the participants were iden-tical, except for 10 discrepancies. One involved a singlebase located on genome termini where, for Illumina

MiSeq data, Labs 1 and 3 called the nucleotide “W” (A/T) and Lab 2 assigned the nucleotide “T”. Nine discrep-ancies were indels (8 of which located within coding re-gions and 1 within an intergenic region) present in theconsensus sequence produced by Lab 2 based on IonProton™ data. This outcome reflects the results obtainedby this laboratory for all the other samples analyzed.The agreement between the different BI pipelines wasassessed also by comparing the consensus sequences ofr23/75 against the sequence of the respective encodingplasmid (p23/75) obtained by Sanger sequencing andused as a gold standard reference. Apart from the 10 dis-crepancies described above, all the consensus sequenceswere identical to the reference, suggesting that the BIpipelines used by the participants were basically correct.For Illumina MiSeq data, accuracy of Lab 1 and 3 was99.991%, while Lab 2 achieved 100% (mean value99.994%). For Ion Proton™ data, Lab 1 and 3 pipelineswere more accurate (100%), while Lab 2 accuracy was99.919% (mean value 99.973%). Overall accuracy was99.983%.

Explanatory analysis of discrepanciesTo discover the origin of the inconsistencies among thesequences produced by the participants and referring tothe same strain, we performed an explanatory analysison a selection of 50 discrepancies representative of theobserved cases and randomly chosen. The subset in-cluded indels and SNPs within intergenic and coding re-gions. Indels located within the coding regions wereexcluded from this analysis, as we have previously ob-served that such inconsistencies are due to a combin-ation of factors, i.e. limits of the Ion Proton™ technologyin resolving sequences rich in homopolymers coupledwith a technical limitation of Lab 2 BI pipeline probablyrelated to the variant caller adopted.When laboratories were questioned individually for

the reasons of their choices in nucleotide assignment,28% were ascribable to alignment, 43% to variant calling,15% to manual curation and 14% to de novo assembly(Fig. 3 a). However, the comparative analysis of thelaboratory scores, aimed at the identification of the causeof the discrepancies, revealed that the majority of them(40 out of 50) were due to manual curation of the con-sensus sequence (Fig. 3 b). Among these, 19 out of 40(reported in Fig. 3 b as discrepancies “A”) arose when atleast one participant assigned a particular nucleotideduring manual curation of step iii). Additional 19/40 dis-crepancies (reported in Fig. 3 b as discrepancies “B”) oc-curred after manual curation during step iii) by the threelaboratories as well as after de novo assembly by Lab 1during step iv). Two discrepancies out of 40 (reported inFig. 3 b as discrepancies “C”) arose after de novo assem-bly of Lab 1 during step iv). Finally, the remaining 10

Zamperin et al. Virology Journal (2019) 16:140 Page 8 of 13

discrepancies were due to the variant calling of step ii)performed by the three groups (n = 9) or to the align-ment of step i) (n = 1), which comprises data cleaning,reference choice and alignment tools/QC metrics.As the manual curation appeared to be the major source

for discrepancies, we explored the sites involved and investi-gated whether such inconsistencies could be resolved withsimple adjustments. Twenty-six discrepancies among “A”and “B” involved degenerate bases. In detail, one laboratoryreported all the nucleotide variants observed in the consen-sus sequence, while the other participants selected only oneeligible nucleotide. The adoption of a unique threshold valuefor minimum nucleotide frequency, as well as the setting ofcommon criterion for manual curation of sites where morethan one nucleotide is observed, are proposed as possible so-lutions to obtain consistent results. The two discrepancies“C” were represented by indels found during manual cur-ation of steps iii) and iv). Such indels were missed by one la-boratory that did not check the consensus against the rawdata during the manual curation step. The implementationof this operation or, alternatively, the adoption of differentvariant calling tools might be used to solve this type of dis-crepancy. Finally, inconsistencies among group “A” involvingboth indels (n = 7) and SNPs (n = 5) were due to a variety ofreasons intrinsic to the different BI pipelines so that nopossible shared solution could be proposed to resolve them.

DiscussionReliability of viral genomes generated through NGS is of ut-most importance both for diagnosticians and researchersaiming to genetically characterize infectious agents. NGS isa complex multifaceted process where quality assurance ofsequence data requires a multi-step approach involvingboth the wet- and the dry-bench parts. Nowadays, PTs ofthe entire NGS workflow are increasingly being adopted inthe clinical diagnostics panorama [8, 31–33]. Importantly,the accomplishment of guidelines established by several in-stitutions and working groups to assure quality standardsof NGS data is becoming an important pre-requisite toguarantee laboratory results [7, 34–39].Although sequence data obtained with NGS are largely

used for a variety of applications also in veterinary infec-tiology [4–10], the implementation of PTs in this field isstill at its infancy. In this work, we attempted for thefirst time to use proficiency testing in veterinary medi-cine for the generation of consensus sequences of twoloss-making salmonid viruses, i.e. VHSV and IHNV,based on real virological data. During this exercise, weassessed the comparability of the BI analysis, regardlessof sample preparation procedures, libraries synthesisprotocols and sequencing platforms. Indeed, althoughthe evaluation of the whole NGS process by means ofPTs has indisputable advantages, challenging individual

Fig. 3 Explanation of inconsistencies. a: The histogram shows the total number of justifications for nucleotide assignment given by participantsfor the subset of 50 discrepancies randomly chosen. Justifications are represented by the BI step to which an explanation is ascribed. b: A piechart showing the source distribution of the discrepancy subset; “manual curation” is further divided into three subgroups, depending from thereason behind the discrepancy itself: “A” arisen when at least one participant assigned a nucleotide during manual curation; “B” arisen aftermanual curation as well as after de novo assembly during step iv); “C” arisen after de novo assembly of step iv)

Zamperin et al. Virology Journal (2019) 16:140 Page 9 of 13

assay components might be helpful to better highlightcritical points of each analytical step [40]. The usefulnessof this approach appears clearer when applied to assessthe consistency of diagnostic results among institutionsthat outsource sequencing while performing the BIanalysis in-house, or use diverse NGS technologies andBI pipelines. This is the case of the Novimark project,whose ultimate objective is the identification of virulencemarkers of VHSV and IHNV through WGS. Motivatedby the need to rely on fully comparable sequence datagenerated by different institutions, we organized a profi-ciency test for BI analysis by sharing the same set of rawdata among three laboratories performing NGS, namelyLab 1, 2 and 3. The feasibility of the PT for the evalu-ation of the post-sequencing analytical steps was firstdemonstrated by two recent studies applied to onco-logical diagnosis. Duncavage et al. (2016) [41] used elec-tronic reference data files by editing already existingFASTQ records to introduce variants. In contrast, Davieset al. (2016) [8] used real data obtained from clinicalspecimens. Both studies evaluated the calling of clinicalvariants by comparing data provided by laboratoriesusing the same BI pipeline. This impeded the evaluationof results variability deriving from the use of alternativeapproaches for variant calling, alignment and annotation.In our PT, such a limitation was circumvented by allow-ing laboratories to adopt their own BI pipeline on acommon set of raw data. This enabled to assess the per-formances of each BI pipeline independently from thewet-bench part of sample preparation, thus permittingthe identification of discrepancies at consensus level en-tirely due to the computational phase. A similar ap-proach has recently been used also by Brinkmann et al.(2019) [10], although in their PT the dataset was artifi-cial and in silico simulated. In our exercise we used realdata derived from virus isolates, which realistically repre-sent the potential issues that might arise during BI ana-lysis and add another key point that makes our workunique. However, there are also some limitations to ourstudy. The most significant flaw is likely due to our ap-proach in evaluating the differences between the BI pipe-lines only at the consensus level. This means that wewere actually unable to perform a finer analysis based onthe variant allele frequency (VAF) found by each partici-pant, thus underestimating the limitations specific ofeach BI pipeline. Besides, it is worth mentioning that theassessment of VAF and its implications in the study ofviral quasi species would require a much higher sequen-cing power (i.e. deep sequencing) that we were unable toaccomplish in our work. As reviewed in Quiñones-Mateu et al. (2014) [42], this approach is being increas-ingly used in diagnostic laboratories, given the signifi-cance of VAF in human clinical virology [43]. It wouldbe of great interest to implement this practice on a

routine basis also for animal viruses, where the selectionat viral population level of signatures implicated inimmune evasion and host jump are particularlyrelevant [44–48].Our data showed that the manual curation of the con-

sensus sequence appeared to account for the majority ofinconsistent results among laboratories, while minor rea-sons were alignment and variant calling. We thereforeconcluded that manual curation requires harmonizationand should be carried out with more attention to ensurethe generation of genome sequences fully comparableamong laboratories. In particular, as the nucleotide as-signment during manual curation might be driven by di-verse sources (e.g. re-inspection of raw data, de novoassembly, assessment of the reading frame within codingregions, etc.), it is important to establish common cri-teria to prioritize the importance of these sources inguiding manual edit. In detail, the adoption of the denovo assembly step and the attribution of ambiguousnucleotides turned out to be the major cause of hetero-geneous results. Although we have suggested adjustmentstrategies to harmonize sequence data (e.g. commonthreshold values for degenerated nucleotides), the estab-lishment of unique guidelines for BI analysis was far be-yond the scope of our work and certainly requires abigger effort, taking into account also specific purposesand applications of the NGS assay. For the sake of com-pleteness, however, we must recall once again that man-ual curation is not the only source of discrepanciesamong laboratories. In fact, indels predicted by Lab 2when analyzing Ion Proton™ data, as well as SNPs atgenome termini observed for Illumina MiSeq data, mightbe attributable more likely to issues during variant call-ing rather than manual curation of the final consensus.Van Borm and collaborators (2016) [9] have recently

identified possible sources of error and bias along theNGS workflow. In their review, they have also proposedsome primary guiding principles for the analytical valid-ation of NGS methods applied to animal infectious dis-eases, based on the OIE (World Organization forAnimal Health) recommendations [49, 50]. While ourexercise as such was not designed to assess assayanalytical sensitivity and specificity, we were able toevaluate inter-laboratory reproducibility, defined as theconsistency among consensus sequences produced byLab 1, 2 and 3 under different analytical procedures. Intotal, the exercise yielded 526 discrepancies, resulting inhigh concordance among laboratories (99.94% reprodu-cibility). The analysis of a calibrator sample and thecomparison of the consensus sequences generated withits reference genome obtained by Sanger allowed us toassess that the BI pipelines were accurate. Overall, re-producibility and accuracy values herein obtained are inagreement with a previous report from Kozyreva et al.

Zamperin et al. Virology Journal (2019) 16:140 Page 10 of 13

(2017) [30], and can be used as a baseline for future PTson WGS applied to veterinary infectivology.The sequencing of the calibrator with both Illumina

MiSeq and Ion Proton™ enabled us to compare the BIpipelines also in respect to the sequencing technologies,which are both characterized by strengths and weaknesses[51–55]. Ion Proton™ appears more efficient in terms ofturnaround time and costs (≤4 h; 80$/Gb) if compared toIllumina MiSeq (24–56 h; 110–212$/Gb). Although boththe sequencer machines produce short sequences(≤400 bp), Illumina MiSeq yields high quality reads(0.1% substitution error) of fixed lengths, while IonProton™ generates reads within a wide lengths range andof lower quality (1% substitution error). Besides, IonProton™ also suffers from an intrinsic high rate ofincorrect basecall in homopolymer regions. Despite morereference samples are needed to properly compare thesequencing technologies adopted in this study, ourobservations confirm that Illumina MiSeq appears moreaccurate, while indels in homopolymer regions represent amajor issue for Ion Proton™.

ConclusionsIn summary, we successfully applied a PT test for NGSBI analysis to animal viruses, which turned out to bevalid also outside the human and microbiological clinicalcontext where such exercises have already proved theirefficacy. Our results highlight that the manual curationappears the most critical step affecting assay reproduci-bility, and suggest that a major effort should be made inthe future to harmonize analytical steps of the BI pipe-line. Indeed, the PT herein presented was useful for thepurposes of the Novimark project and, when possible,we recommend implementing ring trials for NGS toguarantee sequence data comparability among differentlaboratories.

Supplementary informationSupplementary information accompanies this paper at https://doi.org/10.1186/s12985-019-1223-8.

Additional file 1. List of viruses used for the BI PT. For all the viruses wehave reported the laboratory performing wet-bench part, the virial spe-cies, and the NGS technology used.

Additional file 2. Primers used to Sanger sequence the vectorcontaining the VHSV 23/75 full-length cDNA (p23–75). The nucleotide po-sitions refer to the VSHV sequence under the GenBank acc. no. FN665788.

Additional file 3. BI pipelines. Three diagrams show a schematicrepresentation of the BI pipeline for consensus sequence generationadopted in each laboratory.

Additional file 4. Distribution of discrepancies per sample. For each rawdata, the number of inconsistencies found is expressed as stackedcolumns. Genome localization is marked with red, blue and green bars(CDS, intergenic regions and genome termini, respectively); SNPs andindels are marked by lighter and darker shades of the corresponding

color, respectively. A horizontal double gray bar points the sequencingtechnology and the viral species for each raw data.

Additional file 5. Availability of sequence data. For each sample, wehave reported the submitter laboratory, the SRA accession number andthe GenBank accession number.

AbbreviationsBI: Bioinformatics; CDS: Coding sequence; CPE: Cytopathic effect; d/s: Differences per sample; ds-cDNA: Double stranded complementary DNA;EPC: Epithelioma papulosum cyprini; FTP: File Transfer Protocol;IHNV: Infectious haematopoietic necrosis virus; Indels: Insertions/deletions;MOI: Multiplicity of infection; NGS: Next generations sequencing;PT: Proficiency test; QC: Quality control; qPCR: Quantitative polymerase chainreaction; RNA: Ribonucleic acid; SNPs: Single nucleotide polymorphisms;SRA: Sequence read archive; VAF: Variant allele frequency; VHSV: Viralhaemorrhagic septicaemia virus; WGS: Whole genome sequencing

AcknowledgementsThe authors thank Michel Brémont and Stéphane Biacchesi (INRA) forproviding the calibrator sample. Isabella Monne, Anna Toffan and MicheleGastaldelli are acknowledged for fruitful discussions. The authors are gratefulto Paola Mozzi and Francesca Ellero for graphic assistance and manuscriptediting.

Authors’ contributionsVP conceived the study. VP, GZ, YB and IC developed the experimentaldesign. IC, MA, AC and EV propagated and sequenced viral isolates. GZ, DS,IC, DR and PL performed the BI analysis. GZ elaborated data collected. GZ,VP, DR, IC and YB interpreted the results. GZ and VP wrote the manuscript.All authors read and approved the final manuscript.

FundingThe present work was developed within the framework of the NovimarkProject (G88F13000660001) funded by the Animal Health and Welfare(Anihwa) ERA-Net Consortium and the UK’s Department for Environment,Food and rural Affairs (Defra) under contract C7277B. Gianpiero Zamperin ac-knowledges support by the Italian Ministry of Health through grant RC IZSVE08/2016.

Availability of data and materialsIllumina MiSeq and Ion Proton™ raw data were submitted to the NCBISequence Read Archive (SRA; www.ncbi.nlm.nih.gov/Traces/sra/). We havedeposited in GenBank (https://www.ncbi.nlm.nih.gov/genbank/) only theconsensus sequences generated by the laboratory performing thesequencing. Consensus sequences generated from laboratories notperforming the sequencing are available upon request to the authors. BothSRA and GenBank accession numbers are listed in Additional file 5.

Ethics approval and consent to participateNot applicable.

Consent for publicationNot applicable.

Competing interestsThe authors declare that they have no competing interests. The content ofthis article reflects only the authors’ views and the ERA-Net Consortium isnot liable for any use that may be made of the information containedtherein.

Author details1Department of Comparative Biomedical Sciences, Istituto ZooprofilatticoSperimentale delle Venezie (IZSVe), viale dell’Università 10, 35120 Legnaro(PD), Italy. 2French Agency for Food, Environmental and Occupational Health& Safety (ANSES), Ploufragan-Plouzané-Niort Laboratory, Viral Genetics andBiosecurity Unit, 22440 Ploufragan, France. 3Bretagne Loire University, placePaul Ricoeur CS 54417, 35044 Rennes, France. 4Centre for Environment,Fisheries and Aquaculture Science (CEFAS), Barrack Road, The NotheWeymouth, Dorset DT4 8UB, UK. 5European Union Reference Laboratory forFish and Crustacean Diseases, DTU aqua, Kemitorvet 202, 2800 Kgs. Lyngby,

Zamperin et al. Virology Journal (2019) 16:140 Page 11 of 13

Denmark. 6French Agency for Food, Environmental and Occupational Health& Safety (ANSES), Ploufragan-Plouzané-Niort Laboratory, Viral Diseases in FishUnit, 29280 Plouzané, France.

Received: 17 July 2019 Accepted: 16 September 2019

References1. Deurenberg RH, Bathoorn E, Chlebowicz MA, Couto N, Ferdous M, García-

Cobos S, et al. Reprint of “Application of next generation sequencing inclinical microbiology and infection prevention.” J Biotechnol. 2017;250:2–10.Available from: https://doi.org/10.1016/j.jbiotec.2016.12.022.

2. Belák S, Karlsson OE, Leijon M, Granberg F. High-throughput sequencing inveterinary infection biology and diagnostics. Rev Sci tech. 2013;32:893–915.Available from: http://www.ncbi.nlm.nih.gov/pubmed/24761741.

3. Lecuit M, Eloit M. The diagnosis of infectious diseases by whole genomenext generation sequencing: a new era is opening. Front Cell InfectMicrobiol. 2014;4. Available from: http://journal.frontiersin.org/article/10.3389/fcimb.2014.00025/abstract .

4. Granberg F, Bálint Á, Belák S. Novel technologies applied to the nucleotidesequencing and comparative sequence analysis of the genomes ofinfectious agents in veterinary medicine. Rev Sci Tech. 2016;35:15–42.Available from: .http://doc.oie.int:8080/dyn/portal/index.seam?page=alo&aloId=33690.

5. Van Borm S, Belák S, Freimanis G, Fusaro A, Granberg F, Höper D, et al.Next-Generation Sequencing in Veterinary Medicine: How Can the MassiveAmount of Information Arising from High-Throughput TechnologiesImprove Diagnosis, Control, and Management of Infectious Diseases?Methods Mol Biol. 2015;415–36. Available from: http://link.springer.com/10.1007/978-1-4939-2004-4_30.

6. Oliver GR, Hart SN, Klee EW. Bioinformatics for clinical next generationsequencing. Clin Chem. 2015;61:124–35. Available from: http://www.clinchem.org/cgi/doi/10.1373/clinchem.2014.224360.

7. Gargis AS, Kalman L, Bick DP, da Silva C, Dimmock DP, Funke BH, et al.Good laboratory practice for clinical next-generation sequencing informaticspipelines. Nat Biotechnol. 2015;33:689. Available from: https://doi.org/10.1038/nbt.3237.

8. Davies KD, Farooqi MS, Gruidl M, Hill CE, Woolworth-Hirschhorn J, Jones H,et al. Multi-Institutional FASTQ File Exchange as a Means of ProficiencyTesting for Next-Generation Sequencing Bioinformatics and VariantInterpretation. J Mol Diagnostics. 2016;18:572-9. Available from: https://doi.org/10.1016/j.jmoldx.2016.03.002.

9. Van Borm S, Wang J, Granberg F, Colling A. Next-generation sequencingworkflows in veterinary infection biology: towards validation and qualityassurance. Rev Sci Tech. 2016;35:67–81. Available from: http://doc.oie.int:8080/dyn/portal/index.seam?page=alo&aloId=33698.

10. Brinkmann A, Andrusch A, Belka A, Wylezich C, Hoper D, Pohlmann A, et al.Proficiency testing of virus diagnostics based on bioinformatics analysis ofsimulated in Silico high-throughput sequencing data sets. J Clin Microbiol.2019;57. Available from: https://doi.org/10.1128/JCM.00466-19.

11. Genus: Novirhabdovirus. Available from: https://talk.ictvonline.org/ictv-reports/ictv_online_report/negative-sense-rna-viruses/mononegavirales/w/rhabdoviridae/796/genus-novirhabdovirus.

12. Wolf K, Quimby MC. Established Eurythermic Line of Fish Cells in vitro.Science. 1962;135:1065–1066. Available from: http://www.sciencemag.org/cgi/doi/10.1126/science.135.3508.1065.

13. Fijan N, Sulimanović D, Bearzotti M, Muzinić D, Zwillenberg LO, ChilmonczykS, et al. Some properties of the Epithelioma papulosum cyprini (EPC) cellline from carp Cyprinus carpio. Ann l’Institut Pasteur / Virol. 1983;134:207–220. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0769261783800604.

14. Biacchesi S, Lamoureux A, Merour E, Bernard J, Bremont M. LimitedInterference at the Early Stage of Infection between Two RecombinantNovirhabdoviruses: Viral Hemorrhagic Septicemia Virus and InfectiousHematopoietic Necrosis Virus. J Virol. 2010;84:10038–10050. Available from:http://jvi.asm.org/cgi/doi/10.1128/JVI.00343-10.

15. Bolger AM, Lohse M, Usadel B. Trimmomatic: a flexible trimmer for Illuminasequence data. Bioinformatics. 2014;30:2114–20. Available from: https://academic.oup.com/bioinformatics/article-lookup/doi/10.1093/bioinformatics/btu170.

16. Langmead B, Salzberg SL. Fast gapped-read alignment with bowtie 2. NatMethods. 2012;9:357–9. Available from: http://www.nature.com/articles/nmeth.1923.

17. Li H, Durbin R. Fast and accurate long-read alignment with Burrows-Wheeler transform. Bioinformatics. 2010;26:589–595. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2828108&tool=pmcentrez&rendertype=abstract.

18. Chevreux B, Wetter T, Suhai S. Genome sequence assembly using tracesignals and additional sequence information. Comput Sci Biol Proc Ger ConfBioinforma. 1999:45–56.

19. Nurk S, Bankevich A, Antipov D, Gurevich AA, Korobeynikov A, Lapidus A,et al. Assembling single-cell genomes and mini-Metagenomes fromchimeric MDA products. J Comput Biol. 2013;20:714–37. Available from:http://www.liebertpub.com/doi/10.1089/cmb.2013.0084.

20. Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic local alignmentsearch tool. J Mol Biol. 1990;215:403–410. Available from: http://www.ncbi.nlm.nih.gov/pubmed/2231712.

21. Li H, Handsaker B, Wysoker A, Fennell T, Ruan J, Homer N, et al. Thesequence alignment/map format and SAMtools. Bioinformatics. OxfordUniversity Press. 2009;25:2078–2079. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2723002&tool=pmcentrez&rendertype=abstract.

22. Seemann T. Prokka: rapid prokaryotic genome annotation. Bioinformatics.2014;30:2068–9 Available from: https://academic.oup.com/bioinformatics/article-lookup/doi/10.1093/bioinformatics/btu153.

23. Seemann T. Snippy: fast bacterial variant calling from NGS reads. 2015.Available from: https://github.com/tseemann/snippy.

24. McKenna A, Hanna M, Banks E, Sivachenko A, Cibulskis K, Kernytsky A, et al.The Genome Analysis Toolkit: a MapReduce framework for analyzing next-generation DNA sequencing data. Genome Res. 2010;20:1297–1303.Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2928508&tool=pmcentrez&rendertype=abstract.

25. DePristo MA, Banks E, Poplin R, Garimella K V, Maguire JR, Hartl C, et al. Aframework for variation discovery and genotyping using next-generationDNA sequencing data. Nat Genet. 2011;43:491–498. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3083463&tool=pmcentrez&rendertype=abstract.

26. Van der Auwera GA, Carneiro MO, Hartl C, Poplin R, del Angel G, Levy-Moonshine A, et al. From FastQ Data to High-Confidence Variant Calls: TheGenome Analysis Toolkit Best Practices Pipeline. 2013;11.10.1–11.10.33.Available from: http://doi.wiley.com/10.1002/0471250953.bi1110s43.

27. Wilm A, Aw PPK, Bertrand D, Yeo GHT, Ong SH, Wong CH, et al. LoFreq: asequence-quality aware, ultra-sensitive variant caller for uncovering cell-population heterogeneity from high-throughput sequencing datasets.Nucleic Acids Res. 2012;40:11189–201. Available from: https://academic.oup.com/nar/article/40/22/11189/1152727.

28. Milne I, Bayer M, Cardle L, Shaw P, Stephen G, Wright F, et al. Tablet--nextgeneration sequence assembly visualization. Bioinformatics. 2010;26:401–2.Available from: https://academic.oup.com/bioinformatics/article-lookup/doi/10.1093/bioinformatics/btp666.

29. Katoh K, Standley DM. MAFFT multiple sequence alignment softwareversion 7: improvements in performance and usability. Mol Biol Evol. 2013;30:772–80. Available from: https://academic.oup.com/mbe/article-lookup/doi/10.1093/molbev/mst010.

30. Kozyreva VK, Truong C-L, Greninger AL, Crandall J, Mukhopadhyay R,Chaturvedi V. Validation and Implementation of Clinical LaboratoryImprovements Act-Compliant Whole-Genome Sequencing in the PublicHealth Microbiology Laboratory. J Clin Microbiol. 2017;55:2502–2520.Available from: http://jcm.asm.org/lookup/doi/10.1128/JCM.00361-17.

31. Merker JD, Devereaux K, Iafrate AJ, Kamel-Reid S, Kim AS, Moncur JT, et al.Proficiency Testing of Standardized Samples Shows Very HighInterlaboratory Agreement for Clinical Next-Generation Sequencing–BasedOncology Assays. Arch Pathol Lab Med. 2018;143(4):463–471. Available from:https://doi.org/10.5858/arpa.2018-0336-CP.

32. Zhong Q, Wagner U, Kurt H, Molinari F, Cathomas G, Komminoth P, et al.Multi-laboratory proficiency testing of clinical cancer genomic profiling bynext-generation sequencing. Pathol Res Pract. 2018;214(7):957–963.Available rom: https://doi.org/10.1016/j.prp.2018.05.020.

33. Hendriksen RS, Pedersen SK, Larsen MV, Pedersen JN, Lukjancenko O, KaasRS, et al. The Proficiency Test (Pilot) Report of the Global Microbial Identifier(GMI) Initiative, Year 2014. Kgs. Lyngby: National Food Institute, Technical

Zamperin et al. Virology Journal (2019) 16:140 Page 12 of 13

University of Denmark.2015. Available from: https://orbit.dtu.dk/files/126068343/Report_The_procifiency_test_pilot_report_of_the_Global_Microbial_Identifier_GMI_initiative_year_2014.pdf.

34. Rehm HL, Bale SJ, Bayrak-Toydemir P, Berg JS, Brown KK, Deignan JL, et al.ACMG clinical laboratory standards for next-generation sequencing. GenetMed. 2013;15:733–47 Available from: http://www.nature.com/articles/gim201392.

35. Aziz N, Zhao Q, Bry L, Driscoll DK, Funke B, Gibson JS, et al. College ofAmerican Pathologists’ laboratory standards for next-generation sequencingclinical tests. Arch Pathol Lab Med. 2015;139:481–93. Available from: http://www.archivesofpathology.org/doi/10.5858/arpa.2014-0250-CP.

36. Jennings LJ, Arcila ME, Corless C, Kamel-Reid S, Lubin IM, Pfeifer J, et al.Guidelines for validation of next-generation sequencing–based oncologypanels: a joint consensus recommendation of the Association for MolecularPathology and College of American pathologists. J Mol Diagnostics. 2017;19(3):341–365. Available from: https://doi.org/10.1016/j.jmoldx.2017.01.011.

37. Roy S, Coldren C, Karunamurthy A, Kip NS, Klee EW, Lincoln SE, et al.Standards and guidelines for validating next-generation sequencingbioinformatics pipelines: a joint recommendation of the Association forMolecular Pathology and the College of American Pathologists. J MolDiagnostics. 2018;20(1):4–27. Available from: https://doi.org/10.1016/j.jmoldx.2017.11.003.

38. Gargis AS, Kalman L, Lubin IM. Assuring the quality of next-generationsequencing in clinical microbiology and public health laboratories. J ClinMicrobiol. 2016;54:2857–2865. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27510831.

39. Rossen JWA, Friedrich AW, Moran-Gilad J, ESCMID study Group for Genomicand Molecular Diagnostics (ESGMD). Practical issues in implementingwhole-genome-sequencing in routine diagnostic microbiology. ClinMicrobiol infect. 2018;24:355–360. Available from: http://www.ncbi.nlm.nih.gov/pubmed/29117578.

40. Schrijver I, Aziz N, Jennings LJ, Richards CS, Voelkerding K V., Weck KE.Methods-Based Proficiency Testing in Molecular Genetic Pathology. J MolDiagnostics. 2014;16:283–287. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1525157814000348.

41. Duncavage EJ, Abel HJ, Merker JD, Bodner JB, Zhao Q, Voelkerding KV, et al.A model study of in silico proficiency testing for clinical next-generationsequencing. Arch Pathol Lab Med. 2016t;140(10):1085–91. Available from:https://doi.org/10.5858/arpa.2016-0194-CP.

42. Quinones-Mateu ME, Avila S, Reyes-Teran G, Martinez MA. Deep sequencing:becoming a critical tool in clinical virology. J Clin Virol Netherlands. 2014;61:9–19. Available from: https://doi.org/10.1016/j.jcv.2014.06.013.

43. Kasibhatla SM, Waman VP, Kale MM, Kulkarni-Kale U. Analysis of Next-generation Sequencing Data in Virology - Opportunities and Challenges. In:Kulski JK, editor. Next Gener Seq. Rijeka: IntechOpen. 2016. Available from:https://doi.org/10.5772/61610.

44. Borucki MK, Chen-Harris H, Lao V, Vanier G, Wadford DA, Messenger S, et al.Ultra-deep sequencing of intra-host rabies virus populations during cross-species transmission. PLoS Negl Trop Dis. 2013;7(11):e2555. Available from:https://doi.org/10.1371/journal.pntd.0002555.

45. Varble A, Albrecht RA, Backes S, Crumiller M, Bouvier NM, Sachs D, et al.Influenza a virus transmission bottlenecks are defined by infection routeand recipient host. Cell Host Microbe. 2014;16:691–700. Available from:https://doi.org/10.1016/j.chom.2014.09.020.

46. Borucki MK, Allen JE, Chen-Harris H, Zemla A, Vanier G, Mabery S, et al. Therole of viral population diversity in adaptation of bovine coronavirus to newhost environments. PLoS One. 2013;8:e52752. Available from: https://doi.org/10.1371/journal.pone.0052752.

47. Dridi M, Rosseel T, Orton R, Johnson P, Lecollinet S, Muylkens B, et al. Next-generation sequencing shows West Nile virus quasispecies diversificationafter a single passage in a carrion crow (Corvus corone) in vivo infectionmodel. J Gen Virol. 2015;96:2999–3009. Available from: https://doi.org/0.1099/jgv.0.000231.

48. Schonherz AA, Lorenzen N, Guldbrandtsen B, Buitenhuis B, Einer-Jensen K.Ultra-deep sequencing of VHSV isolates contributes to understanding therole of viral quasispecies. Vet Res. 2016;47:10. Available from: https://doi.org/10.1186/s13567-015-0298-5.

49. OIE Terrestrial Manual, Chapter 1.1.6. Principles and methods of validation ofdiagnostic assays for infectious diseases. OIE Man Diagnostic Test VaccinesTerr Anim. 2004.

50. OIE Terrestrial Manual, Chapter 1.1.7. Standards for high throughputsequencing, bioinformatics and computational genomics. OIE ManDiagnostic Test Vaccines Terr Anim. 2004.

51. Loman NJ, Misra R V, Dallman TJ, Constantinidou C, Gharbia SE, Wain J,et al. Performance comparison of benchtop high-throughput sequencingplatforms. Nat Biotechnol. 2012;30:434–439. Available from: https://doi.org/10.1038/nbt.2198.

52. Quail MA, Smith M, Coupland P, Otto TD, Harris SR, Connor TR, et al. A taleof three next generation sequencing platforms: comparison of Ion Torrent,Pacific Biosciences and Illumina MiSeq sequencers. BMC Genomics. 2012;13:341. Available from: https://doi.org/10.1186/1471-2164-13-341.

53. Reuter JA, Spacek D V, Snyder MP. High-throughput sequencingtechnologies. Mol cell. 2015;58:586–597. Available from: https://doi.org/10.1016/j.molcel.2015.05.004.

54. Goodwin S, McPherson JD, McCombie WR. Coming of age: ten years ofnext-generation sequencing technologies. Nat Rev Genet. 2016;17:333–51.Available from: https://doi.org/10.1038/nrg.2016.49.

55. Buermans HPJ, den Dunnen JT. Next generation sequencing technology:Advances and applications. Biochim Biophys Acta - Mol Basis Dis. 2014;1842:1932–41. Available from: https://linkinghub.elsevier.com/retrieve/pii/S092544391400180X.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.

Zamperin et al. Virology Journal (2019) 16:140 Page 13 of 13