design and evaluation of a scalable and portable ... · pdf file˝back forw˛ design and...

�back forw�Design and Evaluation of a Scalable and Portable,

Reconfigurable Computing Implementation of BLAST

Parag [email protected]

Computer Systems Design Lab (CSDL)Information and Telecommunication Technology Center (ITTC)

University Of Kansas03/13/2006

Parag Beeraka 1

mailto:[email protected]

�back forw�

Outline

➭ Introduction/Motivation

➭ Background: Reconfigurable Computing (RC) and BLAST

➭ Analysis

➭ RC Implementation

➭ Results

➭ Conclusion & Future Work

Parag Beeraka 2

�back forw�

Motivation

Parag Beeraka 3

�back forw�

BLAST

➭ similarity search for biological databases

➭ based on heuristics

➭ successful for several reasons

❏ results are statistically characterized

❏ source is available

❏ runs on many machines (desktops and up)

➭ molecular biologists use it as an abstract tool

∗ BLAST — Basic Local Alignment Search Tool

Parag Beeraka 4

�back forw�

BLAST Similarity Search

one way BLAST is commonly used:

➭ researcher sequences the genome of a new organism

➭ want to know the function of a particular gene, but wet labtests are expensive (in time and money)

➭ search against known gene functions does preliminary screen-ing

Parag Beeraka 5

�back forw�

Problem Size vs Processor and I/O Performance

uP 59% CAGR

I/O Perf. 10% CAGR

(10 year) CAGR 77% db size

1e+08

1e+09

1e+10

1e+11

1e+12

1994 1996 1998 2000 2002 2004

num

ber o

f nuc

leot

ides

year

#bases(genbank)mp(x)bio(x)io(x)

Parag Beeraka 6

�back forw�Problem and Possible Solutions

➭ same query will take longer on a workstation next year— how much longer will compound annually

➭ Possible Solutions

❏ Use a cluster of machines which can run in parallel

❍ What about cost in terms of power, I/O subsystemsand networking?

❏ Use Reconfigurable Computing

❍ How difficult it is to port BLAST to this method of com-puting

❍ Is the solution scalable, portable and cost-effective

Parag Beeraka 7

�back forw�

Thesis Question

➭ Is a scalable and cost-effective Reconfigurable Implementationof BLAST feasible which can aid scientists and biologists in a moreproductive way?

Parag Beeraka 8

�back forw�

Background

Parag Beeraka 9

�back forw�

Reconfigurable Computing

➭ Reconfigurable Computing

❏ A form of computing where the hardware has the ability to re-configure based on various functions

❏ Usually use FPGA’s as the hardware

➭ Our goal is to partition BLAST

❏ Software

❏ Hardware as a FPGA core

∗ FPGA — Field Programmable Gate Array

Parag Beeraka 10

�back forw�

Target Hardware

➭ Xilinx Virtex II Pro FPGA

❏ Two PowerPC 405 processors

❏ 13,696 freely available logic slices

➭ 256 MB DDR SDRAM

➭ Compact FLASH

➭ Ethernet

➭ Serial Port

Xilinx ML - 310

Parag Beeraka 11

�back forw�

Standard Linux Virtex II Pro FPGA Configuration

Internals of the Virtex II Pro FPGA

Parag Beeraka 12

�back forw�Basic Idea of Similarity Search

➭ subject database — long string of annotated nucleotidesequences

➭ query — an acquired nucleotide sequence that may have some

similarity to part of a sequence in the subject

➭ task:

find all positions in subject where the query is almost the same;by almost we mean the bases match or there is a tolerable numberof ...

insertions substitutionsdeletions transpositions

Parag Beeraka 13

�back forw�

BLAST Heuristic

➭ use query to build a data structure for lookup

➭ stream subject in sequence-by-sequence

➭ if the base pairs match the query try to expand to the left and rightuntil tolerance is exceeded

➭ if tolerance is met, call it a match and output its position

Parag Beeraka 14

�back forw�

BLAST Example

➭ subject seq : ATGCATGTTTTCGTATGATGAGCTTAT

➭ query seq : ATGCATGTTTTCTATGAAGAGCTT

Score = 26.3 bits (13), Expect = 0.002

Identities = 23/24 (95%), Gaps = 1/24 (4%)

Strand = Plus / Plus

Query: 1 atgcatgttttcgtatgaagagctt 24

|||||||||||| ||||| ||||||

Sbjct: 1 atgcatgttttc-tatgatgagctt 23

Parag Beeraka 15

�back forw�

Previous RC and BLAST Work

➭ K. Logue [Honors Thesis 2000] — simulation of one unit

➭ K. Muriki [MS Thesis 2003, HiCOMB 2004]

❏ simulated a network of cores to improve computational• latency• throughput

❏ implemented one core on PCI-based FPGA card5× slower!

Parag Beeraka 16

�back forw�

Analysis

Parag Beeraka 17

�back forw�

Initial Experiments with BLAST

➭ Ran BLAST on various high-end and low-end machines with var-ious queries and databases

500Mhz 1000Mhz 2400Mhz 3200Mhz& & & &

512MB 1024MB 512MB 2048MBMount type RAM RAM RAM RAM

NFS 152m 48m 77m 17mlocal drive 125m 39m 48m 16m

Parag Beeraka 18

�back forw�

Initial Experiments with BLAST (cont’d)

Ran BLAST on various high-end and low-end machines with variousqueries and databases

1

4

16

64

256

1024

400 800 1200 1600 2000 2400 2800 3200Clock Frequency (MHz)

BLAST Performance v. Processor Clock Freq.

BLAST over NFS

ideal(x)

Exe

cutio

n T

ime

(m)

BLAST over IDE500 MHz512MB RAM

1024MB RAM1100 MHz

3200 MHz2048MB RAM

2400 MHz512MB RAM

Parag Beeraka 19

�back forw�Profiling BLAST

So, need to figure out the most time consuming segment in BLAST..

➭ Added the option ’-pg’ to the gcc compiler when compiling BLAST

➭ Used the tool ’gprof’ to determine the most time consuming seg-ment by running it on the executable

➭ The sub-routine BlastNtWordFinder took 85% of the time

➭ BlastNtWordFinder was further analyzed to determine the linesof code that were the most time consuming

➭ Converted the most time consuming part as a different functioncalled critical code (83%)

Parag Beeraka 20

�back forw�

RC Implementation

Parag Beeraka 21

�back forw�RC Implementation of BLAST

Since ML-310 has ...

➭ Virtex II Pro FPGA’s with PowerPC 405’s can run Linux

➭ Cross compiled BLAST runs on Linux on the PowerPC 405’s

➭ Use Xilinx EDK software to build a complete system

➭ User IP cores can be added to any of the Buses

∗ IP — Intellectual Property∗ EDK — Embedded Development Kit

Parag Beeraka 22

�back forw�

RC Implementation of BLAST

Thought....

➭ Most time consuming segment of BLAST can run on slices on theFPGA as an user IP Core

➭ Access to the user IP core can be achieved through a Linux DeviceDriver

➭ Patch BLAST to add access to the Linux Device Driver

∗ IP — Intellectual Property∗ RC — Reconfigurable Computing

Parag Beeraka 23

�back forw�

RC-BLAST Base System➭ Designed an IP Core and a Lookup

Table to replace the critical code func-tion

➭ RC-BLAST IP Core was added as aMaster - Slave Peripheral to the On- Chip Peripheral Bus

➭ Lookup table was designed such thatit can be loaded onto External Mem-ory

PCI

serial

OPB (32 bits/100 MHz)DIMMPLB (64 bits/100 MHz)

Blast

Bridges

PPC405

DI

RC-BLAST IP Core

Parag Beeraka 24

�back forw�RC-BLAST IP Core

➭ Interfaced to the OPB IPIF interfacewhich interfaces to the OPB Bus

➭ Input FIFO of length 1024 and widthof 32 for writing databases

➭ Output FIFO of length 1024 and widthof 32 for writing match information

➭ Registers to keep track of FIFO counts

➭ Registers to keep track of Databasecount

➭ OPB Slave State Machine

➭ OPB Master State Machine to readdata from external memory

��

��

��

��

DDR

IPIF

OPB_BLAST

BLAST_SLAVE BLAST_MASTER

1024 X

Lookup to DDR Memory

32

BRAM

1024 X

BRAM

32

MASTER

MACHINESTATE

SLAVE

MACHINESTATE

ReadCount

Write Count

Input Output

BUS

ReadCount

Write Count

Count

Offset

Subject

Subject

Parag Beeraka 25

�back forw�RC-BLAST Lookup Table

➭ Table is 2∧16× 64bits

➭ Column 1 - flag bit = 1 if hits are≥ 3

➭ Column 2 - pointer if hits > 3

➭ Column 3 - Query Offset for Hit1



➭ Existing design has no support forhits > 3

flag(1 bit)

flag bit = 0 if #hits <= 3

hits > 3POINTER FOR

else flag bit = 1

# HITS (2 bits)

2^16

3

2

1

0

3 13 16 16 16

48

Query Offset for Hit1 Query Offset for Hit2 Query Offset for Hit3

Parag Beeraka 26

�back forw�RC-BLAST Software

➭ Device Driver

❏ rcblast open() – Initializes the hardware registers

❏ rcblast write lut() – Writes the Lookup table to the externalmemory

❏ rcblast write fifo() – Writes databases to the FIFO’s

❏ rcblast read fifo() –Reads back hit information from the FIFO’s

➭ Patch for BLAST source

❏ init hw()

❏ invoke hw()

Parag Beeraka 27

�back forw�

RC-BLAST Flow

��

��

��

��

false

true

Load "lookup_table"&

Initialize hardware FPGA Slices

For each sequence

blastall −p blastn −i query −d database −o results

critical_code

Input FIFO

Load sequences into

Ouput FIFO information from

Obtain hit

If

First_trip

Linux Device Driver

Similarity matching

against query

Xilinx Vitex II Pro FPGA

BlastNtWordFinder

Parag Beeraka 28

�back forw�

(1) On-Chip Caching of Partial Look-Up

➭ Since Lookup table is in Off-chip memory, IP cores take lot ofclock cycles to access

➭ Used Available BlockRAM’s in the FPGA to cache the first columnof the lookup table

➭ Made appropriate changes to the IP core to add BRAM’s and ap-propriate lookup

➭ Made appropriate changes to the Device Driver to load the BRAM’swith the first column of the Lookup table

Parag Beeraka 29

�back forw�(2) Parallel Design

RC-BLAST IP Core with support for partial lookup➭ RC-BLAST Core with support for

on-chip caching of partial lookupconsumes

❏ 800 logic slices which is 4% ofthe FPGA slices

❏ 36 BRAM’s were consumedwhich is 26% of available

➭ Could replicate only 2 RC-BLASTcore units with support for partiallookup

OPB (32 bits/100 MHz)

PLB (64 bits/100 MHz)

DIMM

Blast BlastBlast Blast Blast

PPC405

I D

Bridges

RC-BLAST IP Core with support for Partial Lookup doesn’t achievemuch parallelism

Parag Beeraka 30

�back forw�

(2) Parallel Design (cont’d)

RC-BLAST IP Core

➭ RC-BLAST Core consumes 800 logicslices which is 4% of the FPGA slices

➭ Rest of the Base System consumes48% of the FPGA slices

➭ Replicated up to 8 RC-BLAST coreunits

OPB (32 bits/100 MHz)

PLB (64 bits/100 MHz)

DIMM

Blast BlastBlast Blast Blast

PPC405

I D

Bridges

RC-BLAST IP Core is more efficient

Parag Beeraka 31

�back forw�Results: Cost Analysis

➭ Compared cost of number of unitsthat can fit in FPGA’s vs Cluster ofprocessors

➭ Considered an average cluster nodeto be 450$

➭ v2p30 - Present FPGA in the TargetHardware

➭ v2p100 - One of the largest FPGA’s

Clearly FPGA’s give more units in terms

of cost 0

3000

6000

9000

12000

15000

18000

21000

24000

27000

30000

0 10 20 30 40 50 60 70 80 90 100C

ost

in d

oll

ars(

$)

Number of BLAST units

Cost Effectiveness of RC−BLAST

v2p100(x)v2p30(x)

cpu(x)

∗ Results are based on RC-BLAST IP core without support for Partial Lookup

Parag Beeraka 32

�back forw�

(Expected) Results: Power Dissipation

➭ Power is definitely a big concernnowadays

➭ Compared power dissipation ofexisting processors with RC-BLASTunits running in the FPGA

Clearly, we would expect FPGA’s dissi-

pate lot lesser power than any of the ex-

isting processors available. 1

100

10000

0 10 20 30 40 50 60 70 80 90 100Po

wer

Dis

sipa

tion

(W) i

n lo

gNumber of Cores

Comparision of RC−BLAST Power Dissipation

xeon(x)pentiumM(x)

athlon(x)opteron(x)

fpga(x)

Parag Beeraka 33

�back forw�

Results: Speedup➭ Number of Lookup’s / second con-

sidered as a measure to calculatespeedup

➭ RC-BLAST runs 10× faster thanjust the Software version of BLASTon the FPGA

➭ RC-BLAST almost same as conven-tional processors

➭ With more RC-BLAST units, we candefinitely achieve more speedup

0

2e+06

4e+06

6e+06

8e+06

1e+07

H/W & S/W on xc2vp30 S/W on xc2vp30 S/W on desktopN

um

ber

of

Lo

ok

up

’s /

sec

myquery vs ecoli.nt

BLAST search between a query and a database

Parag Beeraka 34

�back forw�

Results: Portability

➭ Portability

❏ Added RC-BLAST IP Core as a User IP core to ML-403 Base sys-tem which hosts a Xilinx Virtex IV FPGA

❏ Synthesized to a bit stream without any additional effort

❏ As technology progresses abiding Moore’s Law gives more tran-sistors in a die-

❍ For FPGA’s it is more logic

❍ For processor’s it is more cache

Parag Beeraka 35

�back forw�

Conclusion

➭ Achieved all the design objectives

❏ A design on Reconfigurable Computing resources

❏ Achieved scalability of the design with regards to lowcost, low power and speedup

❏ Developed a portable design

Parag Beeraka 36

�back forw�

Conclusions

➭ Is a scalable and cost-effective Reconfigurable Implementationof BLAST feasible which can aid scientists and biologists in a moreproductive way? Yes

Parag Beeraka 37

�back forw�

Contributions

Along with answering the thesis question .....

➭ Identified the I/O bound problem of BLAST by performing vari-ous experiments

➭ Ported Linux to a Platform FPGA

❏ Developed Cross compiler for PPC405 platform

❏ Developed Device Drivers to access hardware from Linux

➭ Design Extensions

❏ Developed a system where parallelism can be obtained in BLAST

Parag Beeraka 38

�back forw�

Future Work

➭ Change the existing OPB Bus Interface to a PLB Bus interface oran APU interface for the RC-BLAST IP Core

➭ Port RC-BLAST to a Cray XD-1 or a SGI - Altix

➭ Port RC-BLAST to a FPGA based Cluster

➭ Develop software to handle multiple units in an FPGA

➭ Develop RC-BLAST with multiple units without the need for Bus-Arbitration

Parag Beeraka 39

�back forw�

Thank You

Questions !!!!

Parag Beeraka 40

List of Slides

Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

BLAST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

BLAST Similarity Search . . . . . . . . . . . . . . . . . . . . . . . 5

Problem Size vs Processor and I/O Performance 6

Problem and Possible Solutions . . . . . . . . . . . . . . . 7

Thesis Question . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Reconfigurable Computing . . . . . . . . . . . . . . . . . . 10

Target Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Standard Linux Virtex II Pro FPGA Configuration12

Basic Idea of Similarity Search . . . . . . . . . . . . . . . 13

BLAST Heuristic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

BLAST Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Previous RC and BLAST Work . . . . . . . . . . . . . . . . 16

Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Initial Experiments with BLAST . . . . . . . . . . . . . . 18

Initial Experiments with BLAST (cont’d) . . . . . . 19

Profiling BLAST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

RC Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 21

RC Implementation of BLAST . . . . . . . . . . . . . . . 22

RC Implementation of BLAST . . . . . . . . . . . . . . . 23

RC-BLAST Base System . . . . . . . . . . . . . . . . . . . . . . 24

RC-BLAST IP Core . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

RC-BLAST Lookup Table . . . . . . . . . . . . . . . . . . . . . 26

RC-BLAST Software . . . . . . . . . . . . . . . . . . . . . . . . . . 27

RC-BLAST Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

(1) On-Chip Caching of Partial Look-Up . . . . . . 29

(2) Parallel Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

(2) Parallel Design (cont’d) . . . . . . . . . . . . . . . . . . . 31

Results: Cost Analysis . . . . . . . . . . . . . . . . . . . . . . . . 32

(Expected) Results: Power Dissipation . . . . . . . 33

Results: Speedup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Results: Portability . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37

Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39

Thank You . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

design and evaluation of a scalable and portable ... · pdf file˝back forw˛ design and...

Documents