monte carlo simulation for radiotherapy in a distributed computing environment
Post on 05-Jan-2016
54 Views
Preview:
DESCRIPTION
TRANSCRIPT
Monte Carlo simulation for Monte Carlo simulation for radiotherapy in a distributed radiotherapy in a distributed
computing environmentcomputing environment
S. Chauvie2,3, S. Guatelli2, A. Mantero2, J. Moscicki1, M.G. Pia2
CERN1
INFN2
S. Croce e Carle Hospital Cuneo3
Monte Carlo 200518-21 April 2005
Chattanooga, TN, USA
Monte Carlo methods in radiotherapyMonte Carlo methods in radiotherapy
Monte Carlo methods have been explored for years as a tool for precise dosimetry, in alternative to analytical methods
de facto,
Monte Carlo simulation is not used in clinical practice
(only side studies)
The major limiting factor is the speedspeed
The realityThe reality
Treatment planning is performed by means of commercial software
The software calculates the dose distribution delivered to the patient
Open issues
DisadvantagesDisadvantages AdvantagesAdvantages
Commercial systems are based on analytical methodsanalytical methods
Fails in calculate dose in heterogeneities and for small or complex field
Quick responseQuick response
Each treatment planning software is specific to one radiotherapic techniquespecific to one radiotherapic technique
ProjectProjectDevelop a dosimetric system for radiotherapy treatments based on Monte Carlo methods
ProjectProjectDevelop a dosimetric system for radiotherapy treatments based on Monte Carlo methods
Geant4 as Simulation Toolkit
ParallelisationParallelisationAccess to distributed computing resources Access to distributed computing resources
Calculation precision
Quick response
Pilot project: distributed simulation for brachytherapy
Pilot project: distributed simulation for brachytherapy
Explore Geant4-based Monte Carlo simulations in a distributed computing environment
– Parallel execution in a local PC farm– Geographically distributed execution on the GRID
Pilot project based on an existing simulation for brachytherapy
Focus on architectural design– Transparent execution on a single machine, in parallel on a
local farm or on the GRID
Preliminary evaluation of performance
Application to other radiotherapy simulations currently in progress
BrachytherapyBrachytherapy
Simulation of the energy deposited by a radioactive source in a phantom
Requirement from clinical practice: real time response
Bebig Isoseed I-125 source
source
Plan containing the radioactive source
Dose DistributionTalk: “A general purpose dosimetric system for brachytherapy”,20th April, MC 2005, Room 5
Performance in sequential mode
Performance in sequential mode
Endocavitary brachytherapy
1M events
61 minutes
Interstitial brachytherapy
1M events
67 minutes
Superficial brachytherapy
1M events
65 minutes
on an “average” PIII machine
Monte Carlo simulation is not practically conceivable for clinical application, even if more precise
Speed adequate for clinic useSpeed adequate for clinic useSpeed adequate for clinic useSpeed adequate for clinic use
Transparent configuration in sequential or parallel mode
Transparent access to the GRID Transparent access to the GRID through an intermediate software layerthrough an intermediate software layer
Parallelisation
Access to distributed computing resources
Access to distributed computingAccess to distributed computingAccess to distributed computingAccess to distributed computing
speed OK
but expensive hardware investment + maintenance
Hospital LAN
SWITCH
Node01
Node02
Node03
Node04
IMRT
Geant4 Simulation and Anaphe analysis on a dedicated Beowulf ClusterS. Chauvie et al., IRCC Torino, Siena 2002
Access to distributed computingAccess to distributed computingAccess to distributed computingAccess to distributed computing
Alternative strategy
DIANEDIANE
Parallelisation Access to the GRID
Transparent access to a distributed computing environment
Active Workflow Framework for Parallel JobsActive Workflow Framework for Parallel Jobs
Applications run inside an Active Workflow Framework
For applications:– underlying environment is
transparent– code changes to use the framework
are minimal
The Framework provides:– Automatic Communication and
Synchronization of tasks– Error recovery– Optimization
DIANE DIstributed ANalysis EnvironmentDIANE DIstributed ANalysis Environment
prototype for an intermediate layer between applications and the GRID
Parallel cluster processingParallel cluster processing– make fine tuning and customisation easy– transparently using GRID technology– application independentapplication independent
Hide complex details of
underlying technology
Developed by J. Moscicki, CERN
http://cern.ch/DIANE
DIANE architectureDIANE architecture
Master-Worker modelMaster-Worker modelParallel execution of independent tasksVery typical in many scientific applicationsUsually applied in local clusters
R&D in progress forR&D in progress forLarge Scale Master-Large Scale Master-Worker ComputingWorker Computing
Master - Worker ComputingMaster - Worker Computing
Workers are started up and register to Master
Client connects to Master and starts up the job
Master controls the execution, dispatches tasks to Workers and combines the result
Client receives notifications about the current status of the job and collects the final result
Running in a distributed environment
Running in a distributed environment
Not affecting the original code of application– standalone and distributed case is the same codesame code
Good separation of the subsystems– the application does not need to know that it runs in distributed environment– the distributed framework (DIANE) does not need to care about what
actions an application performs internally
The application developer is shielded from the complexity of underlying technology via DIANE
Distributed environmentsDistributed environments
Different distributed environments:
local computing farm GRID
Parallel mode: local cluster / GRIDParallel mode: local cluster / GRID
Both applications have the same computing model– a job consists of a number of independent tasks which may be executed in parallel – result of each task is a small data packet (few kilobytes), which is merged as the
job runs
In a cluster:– computing resources are used for parallel execution– user connects to a possibly remote cluster– input data for the job must be available on the site – typically there is a shared file system and a queuing system– network is fast
GRID computing uses resources from multiple computing centres– typically there is no shared file system– (parts of) input data must be replicated in remote sites– network connection is slower than within a cluster
Development costsDevelopment costsStrategy to minimise the cost of migrating a Geant4 simulation to a distributed environment for users
DIANE Active Workflow framework– provides automatic communication/synchronization mechanisms– application is “glued” to the framework using a small Python module; in most cases
no code changes to the original application are required– load balancing and error recovery policies may be plugged in form of simple python
functions
Transparent adaptation for Clusters/GRIDs, shared/local file systems, shared/private queues
Cost in the runtime phase: – near zero (except for loading networking libraries for the first time)
Development/modification of application code – original source code unmodified – addition of an interface class which binds together application and M-W framework
Interfacing a Geant4 simulation to DIANE Interfacing a Geant4 simulation to DIANE
UML Deployment Diagram for Geant4 applications
Practical example: G4 simulation with analysisPractical example: G4 simulation with analysis
Each task produces a file with histograms
The job result is the sum of histograms produced by tasks
Master-worker model– client starts a job– workers perform tasks and produce histograms– master integrates the results
Distributed Processing for Geant4 Applications– task = N events – job = M tasks – tasks may be executed in parallel – tasks produce histograms/ntuples – task output is automatically combined (add histograms, append ntuples)
Master-Worker Model – Master steers the execution of job, automatically splits the job and merges the results – Worker initializes the Geant4 application and executes macros – Client gets the results
DIANE Prototype and TestingDIANE Prototype and Testing
Scalability tests– 70 worker nodes– 140 milion Geant 4 events
Performance : parallel modePerformance : parallel mode
1M events
4 minutes 34’’
5M events
4 minutes 36’’
1M events
4 minutes 25’’
on up to 50 workers, LSF at CERN, PIII machine, 500-1000 MHz
Performance adequate for clinical application, but…
it is not realistic to expect any hospital to own and maintain a PC farm
Endocavitary brachytherapy
Interstitial brachytherapy
Superficial brachytherapy
preliminary: further optimisation in progress
Parallel mode: distributed resources
Parallel mode: distributed resources
Distributed Geant 4 Simulation:
DIANE framework and generic GRID middleware
GridGrid Wave of interest in grid technology as a basis for “revolution” in e-Science and e-Commerce
An infrastructure and standard interfaces capable of providing transparent access to geographically
distributed computing power and storage space in a uniform way
Ian Foster and Carl Kesselman's book:
”A computational Grid is a hardware and software infrastructure that provides dependable, consistent , pervasive and inexpensive access to
high-end computational capabilities”".
US projectsEuropean projects
Many GRID R&D projects, many related to HEP
Large distributed computing resourceLarge distributed computing resource
Running on the GRIDRunning on the GRIDVia DIANE
Same application code as running on a sequential machine or on a dedicated cluster
– completely transparent to the user
A hospital is not required to own and maintain extensive computing resources to exploit the scientific advantages of Monte Carlo simulation for radiotherapy
Any hospital
– even small ones, or in less wealthy countries, that cannot even small ones, or in less wealthy countries, that cannot afford expensive commercial software systemsafford expensive commercial software systems –
may have access to advanced software technologies and tools for radiotherapy
Traceback from a run on CrossGrid testbed
Traceback from a run on CrossGrid testbed
Current #Grid setup (computing elements):5000 events, 2 workers, 10 tasks (500 events each)
- aocegrid.uab.es:2119/jobmanager-pbs-workq- bee001.ific.uv.es:2119/jobmanager-pbs-qgrid- cgnode00.di.uoa.gr:2119/jobmanager-pbs-workq- cms.fuw.edu.pl:2119/jobmanager-pbs-workq- grid01.physics.auth.gr:2119/jobmanager-pbs-workq- xg001.inp.demokritos.gr:2119/jobmanager-pbs-workq- xgrid.icm.edu.pl:2119/jobmanager-pbs-workq- zeus24.cyf-kr.edu.pl:2119/jobmanager-pbs-infinite- zeus24.cyf-kr.edu.pl:2119/jobmanager-pbs-long- zeus24.cyf-kr.edu.pl:2119/jobmanager-pbs-medium- zeus24.cyf-kr.edu.pl:2119/jobmanager-pbs-short- ce01.lip.pt:2119/jobmanager-pbs-qgrid
Spain
Poland
Greece
Portugal
Resource broker running in Portugal
matchmaking CrossGrid computing elements
Study in progressStudy in progress
Capability of transparent execution of the radiotherapy simulation on the GRID has been demonstrated
Quantitative evaluation of performance speed and stability currently in progress
A comprehensive study will be submitted for publication in the coming weeks
Optimisation of load balancing, error handling and other issues concerning access to distributed resources currently under study
Application to IMRT simulationsApplication to IMRT simulationsDetermine the dose distribution in a phantom generated by the head of a linear accelerator
Requirement from clinical practice: fast response
Without parallelisation:
1010 events
100 CPU days on Pentium IV 3 GHz
Talk: “Geant4 Simulation of an Accelerator Head for Intensity Modulated RadioTherapy”,
19th April, MC 2005, Room 6
Lateral profile6MV, 5x5cm field, 15mm depth
ConclusionsConclusions
Fast performance– parallel processing
Access to geographically distributed computing resources– GRID
Demonstrated with Geant4 simulation applications + DIANE
More information– cern.ch/diane– http://www.ge.infn.it/geant4– www.ge.infn.it/geant4/techtransf– aida.freehep.org
top related