1

Paper Number 097469 1

High performance computing application to address non-point source pollution at a 2

watershed level 3

Chetan Maringanti, 4

Graduate Student, Department of Agricultural and Biological Engineering, Purdue 5 University, West Lafayette, IN. 47906, email: cmaringa@purdue.edu 6

Indrajeet Chaubey, 7

Associate Professor, Department of Agricultural and Biological Engineering and 8 Department of Earth and Atmospheric Sciences, Purdue University, West Lafayette, IN. 9 47906, email: ichaubey@purdue.edu 10

11

Abstract. Best management practices (BMPs) have been proven to effectively reduce the 12

Nonpoint source (NPS) pollution loads from agricultural areas. BMP selection and placement 13

problem needs to be addressed for the placement of BMPs in a watershed at a field scale that 14

would achieve maximum pollution reduction subjected to minimum cost increase for the 15

placement of BMPs in the watershed through optimization techniques. The BMP selection 16

problem is linked with a hydrologic and water quality simulation model to estimate the pollution 17

loads at various locations in the watershed. However, the optimization techniques get very 18

complicated as the size of the watershed gets larger as the design variables increase 19

proportionately. Also, the BMP optimization is linked to the watershed level hydrologic and 20

water quality simulation mode and this makes the problem require very high amount of 21

computation time for the simulations to get to a near optimal set of BMP placement in the 22

2

watershed. In this regard, the high performance computing (HPC) helps a great deal by providing 23

a platform for running the simulations on a multitude of processors rather than a single 24

processor. Unix serves as the de-facto operating system for most of the HPC applications as the 25

architecture allows to run simulations that take a long time to run in much lesser time when 26

compared to the Windows OS machines. In the present work we have used the Soil and Water 27

Assessment Tool (SWAT) model in combination with a genetic algorithm (GA) optimization 28

technique NSGA-II to optimally select and place BMPs at a watershed scale. We have tested the 29

application of the optimization algorithm to reduce sediment, nitrogen, and phosphorus from the 30

L’Anguille River Watershed, Arkansas and pesticide from Wildcat Creek Watershed, Indiana. 31

The SWAT model was compiled in Unix and the optimization routine for BMP selection and 32

placement was coded as an objective function in NSGA-II program in C language. The 33

optimization model was compiled in Unix to optimize the various BMPs that can be placed at a 34

field scale, which is represented as a Hydrologic Response Unit ( HRU) in SWAT. 35

Keywords. High performance computing, BMP, optimization, nonpoint source pollution 36

37

38

39

40

41

42

43

3

Introduction 44

Best management practices (BMP) selection and placement problem consists of finding an 45

optimum solution for BMP placement in a watershed that will achieve the two most important 46

objectives of maximum pollution reduction with a minimum net cost increase for the 47

implementation of BMPs. It is required to optimize the two objectives with the variables being 48

the BMPs at every farm level in the watershed. Optimization algorithms such as genetic 49

algorithms are widely used for BMP selection and placement in a watershed. Most of the 50

optimization models developed are linked dynamically with a watershed level simulation model 51

to estimate the pollutant loads at a watershed scale during the optimization [Arabi et al., 2006; 52

Bekele and Nicklow, 2005; Srivastava et al., 2002]. Presence of the dynamic linkage in the BMP 53

selection and placement models restrict their application to relatively smaller watersheds. Also, 54

the number of possible solutions to be searched during optimization increases exponentially with 55

the number of BMPs considered for implementation due to which a vast number of BMPs were 56

not considered. This problem was addressed through development of a novel methodology that 57

utilizes a database, called as BMP tool, to estimate the pollution effectiveness of any given BMP 58

and therefore removes the requirement of a dynamic linkage with the watershed simulation 59

model during the optimization [Gitau et al., 2006; Maringanti, 2008]. Also, the model for 60

optimization can be run for multiple pollutants and for multiple years of consideration 61

independent to each other and this is a slightly computationally intensive job. More information 62

regarding the model and the method used can be obtained from Maringanti et al., [2009]. In 63

order to validate the applicability of the solutions obtained after optimization using the BMP 64

tool, some solutions need to be picked up from the Pareto-optimal front and evaluated through 65

4

the watershed model to check if similar pollution reductions were obtained. The evaluation of 66

these solutions using the watershed model is again a computationally time intensive job. 67

High performance computing (HPC), that consists of super computers and cluster of processors 68

are used to solve problems that are highly computation intensive. Multiple jobs that are required 69

to accomplish a particular task are distributed to make them run in parallel among multiple 70

processors and therefore decreasing the overall wall clock time for execution. HPC has been 71

used extensively in calibrating hydrologic [Confesor and Whittaker, 2007] and environmental 72

modeling [Vrugt et al., 2006]. As the number of processors increase, the overall computation 73

time is reduced in a range of 50% - 80% [Vrugt et al., 2006]. 74

HPC would greatly help reduce the computation time required to run replicates of the 75

optimization algorithm for multiple pollutants and multiple average years of concern. Also, the 76

computation time required to evaluate the watershed model the solutions picked from the BMP 77

tool output would be highly reduced using HPC. 78

79

Objectives 80

The overall goal of this study is to evaluate the performance of HPC based on a Unix based 81

cluster when compared to the performance of a single processor in windows. In order to achieve 82

this overall goal, the following tasks are performed 83

1) The BMP optimization tool based on a genetic algorithm is run for nitrogen, phosphorus, 84

and sediment along with 5 different averages of pollutant loads (5yr, 10yr, 15yr, 20yr, 85

and 25 yr) in Lincoln Lake Watershed, Arkansas. 86

5

2) The validation of the solutions obtained from the optimization of the BMP optimization 87

tool are used to evaluate the pollutant loads using a watershed model to estimate nitrogen, 88

phosphorus, and sediment in L’Anguille River Watershed; and pesticide in Wildcat Creek 89

Watershed. 90

91

Theoretical Background 92

Multi-objective genetic algorithm (MOGA) 93

94

Genetic algorithms are heuristic based search algorithms, based on Darwin’s theory of evolution, 95

that function well in searching optimum solutions when the search space gets large in the 96

variable and objective function domain. As the BMP selection and placement problem requires 97

two objectives (reduction in pollutant loads and reduction in net cost for placement of BMPs) to 98

be minimized simultaneously, it is required to use a multi-objective genetic algorithm (MOGA) 99

for optimization. Non-dominated sorted genetic algorithm (NSGA-II) (Deb, 1999, 2001; Deb et 100

al., 2002) is one such MOGA that can search a large number of variables and objective functions 101

space to find an optimal solution. The overall computation complexity of the algorithm is 102

O(MN2), which usually is O(MN3) for most of the evolutionary techniques (Deb et al., 2002): 103

where O is ‘order of’, M is the number of objective functions, and N is the population size. Non-104

domination sorting, crowding distance, and elite sets are important properties of the NSGA-II 105

algorithm that ensure that the solutions that are obtained during the optimization have a good 106

spread in the objective function space and the optimization is ensured to reach a Pareto-optimum 107

(Figure 1). This also ensures that the algorithm does not undergo a premature convergence. 108

Crossover and mutation are other genetic operations that are important to generate the future 109

6

offspring based on the parental genetic information and create genetic modification to reach 110

convergence respectively. 111

112

Watershed model (Soil and Water Assessment Tool) 113

The watershed model used in this study was Soil and Water Assessment Tool (SWAT) model. 114

The SWAT is a process based distributed parameter watershed scale simulation model designed 115

for use in gauged as well as ungauged basins to simulate long term effects of various watershed 116

management decisions on hydrology and water quality response (Arnold et al., 1998). The 117

SWAT model divides the watershed into subwatersheds or subbasins based on the outlets 118

selected within the watershed by the user and further divides subbasins into land areas, called 119

hydrologic response units (HRUs), based on land use, management, and soil properties. SWAT 120

performs most of the calculations at HRU level, which are then delivered into the stream present 121

in the particular subbasin and then routed to the downstream subbasin. 122

The input data needed by the model are related to watershed physical characteristics, climate, 123

plant growth, reservoir data (if any), and management practices at HRU level. The typical GIS 124

data needed by the model are digital elevation map (DEM), soil, and land use maps. The climatic 125

input data required by SWAT are precipitation, temperature, solar radiation, relative humidity, 126

and wind on a daily scale from multiple climatic gauge locations. Agricultural activities can be 127

given as input to the model by modifying the management files. SWAT simulates the flow, 128

nutrients, sediment, and chemicals at the subbasin or HRU level. Surface runoff is computed 129

using a modification of the SCS curve number technique (Soil Conservation Service, 1972). 130

Modified Universal Soil Loss Equation (MUSLE) (Williams, 1975) is used by the SWAT model 131

to estimate the soil erosion and sediment yield in the watershed. Nitrogen and phosphorus are 132

7

applied through fertilizer, manure or residue application. Bulk of phosphorus transport takes 133

place in the adsorbed form to the sediment whereas nitrogen is highly dissolvable in water and 134

therefore transported mainly through surface/subsurface flow. The important feature in SWAT is 135

that it aids in modeling the various BMPs (structural and management based) by changing 136

appropriate parameters in the input files of the model. This feature is utilized in the development 137

of BMP tool which estimates the effectiveness of BMPs for a particular pollutant reduction. 138

More details about the development of BMP tool can be obtained from [Maringanti, 2008]. 139

High performance computing facility 140

The Steele supercomputing facility at Purdue University consists of a cluster that is made up of 141

893 Dell dual quad-core computer nodes with an estimate peak performance of 60 teraflops. The 142

machines run on Red Hat Linux operating system with various computational packages installed 143

on each machine. Steele supercomputing facility is the largest among the universities located in 144

the Midwest region of the United States and ranks 104 in the TOP500 list of global 145

supercomputers. The parallel simulations of the various model runs were run on the Steele 146

cluster from which our research group owns 4 nodes (32 processors). 147

Study area and data available 148

Lincoln Lake watershed (LLW) is located within the Illinois River Basin located in Northwest 149

Arkansas. The watershed has a total area of 32 km2 with a mixed land use consisting of pasture, 150

forest, urban residential, urban commercial and water representing 35.84%, 48.6%, 11.9%, 151

1.54% and 2.2% of the watershed area, respectively (Figure 1). The watershed contains more 152

several hundreds of chicken houses. The 303 (d) lists Illinois River an impaired water body with 153

phosphorus being the main cause of impairment. 154

8

L’Anguille river watershed (LRW) is located in the Mississippi delta region of east Arkansas 155

(Figure Error! No text of specified style in document..). The watershed covers an area of 2520 156

km2 and drains the entire stretch (157 km) of the L’Anguille River. LRW is predominantly an 157

agricultural watershed; with more than 60% of the watershed in arable land. The main crops 158

grown in the watershed are rice (26%) and soybean (46%) and represent most (~95%) of the 159

agricultural land in the watershed (Error! Reference source not found.). The L’Anguille River 160

was included in the list of impaired water bodies by the Arkansas Department of Environmental 161

Quality (ADEQ) in 1998 (ADEQ, 2002). Excessive sediment and nutrients are the main source 162

of impairment of the river (ADEQ, 2002). 163

Wildcat Creek watershed (WCW) ( 164

Figure Error! No text of specified style in document..) (USGS 8 digit HUC 05120107) with a 165

drainage area of 1956 km2, located in north central Indiana was used for testing the optimal BMP 166

selection and placement. The watershed is predominately agricultural with 74% row crops (36% 167

Soybean, 38% Corn), 21% pasture, and 3% urban area (NASS, 2001). A high pesticide (atrazine) 168

level from the agricultural areas has degraded the water quality of many watersheds in Indiana 169

(Homes et al., 2001). Various NPS pollution reduction projects are being undertaken in the 170

watershed through funds from the clean water act section 319. These funds, to effectively be 171

utilized to reduce NPS pollutants, need to be combined with a BMP optimal selection and 172

placement tool to choose the best economical and ecological alternative. The proposed multi-173

objective optimization methodology can be applied for the selection and placement of BMPs in 174

Wildcat Creek Watershed for pesticide (atrazine) reduction from areas planted with corn. 175

176

177

9

178

Methodology 179

Figure 5 describes the methodology that is adopted during the computation process. Nitrogen, 180

phosphorus, and sediment are the three pollutants are considered for the LRW. The SWAT 181

model that is used as a pollutant load estimator was simulated for 25 years. Five different annual 182

averages (for 5, 10, 15, 20, and 25 years) were considered to study how the annual pollutant 183

loads are changed due to the averaging process. These 5 averages and 3 different pollutants of 184

concern require 15 different BMP selection and placement model runs to obtain the respective 185

Pareto-optimal solutions after the optimization. These 15 model runs were made to run on 186

Windows machines using a Windows compiled executable and on Steele supercomputer using a 187

Unix compiled executable. The multiple jobs that are required for evaluating the optimization 188

model are distributed into multiple processors on Steele to generate the final Pareto-optimal 189

fronts that provide a range of solutions for the two objective functions. 190

Solutions from the final Pareto-optimal front are picked ensuring that the entire range is covered. 191

N, P, and sediment optimization solutions were used from the LRW and pesticide solutions were 192

used from the WCW. The variables that represent the BMPs to be implemented in each of the 193

HRU are used to modify the input SWAT files and simulate the SWAT model to get the 194

pollutant loads considering the in stream routing processes. The SWAT model used in the 195

simulations was compiled in Unix and the runs are made on the Steele for different possible 196

SWAT input configurations corresponding to the implemented BMPs. 197

198

199

10

200

Results and discussion 201

Table 1 illustrates how the SWAT model computation times vary for the various watersheds 202

considered in the study using both Windows and Unix compiled versions. It is observed that the 203

Unix version of SWAT outperforms the Windows counterpart by being more than 20 times 204

computationally efficient. 205

Output from the SWAT model is used to develop the BMP tool and as an input into the NSGA-II 206

and the optimization run is made with a population size of 800 and run for 20,000 generations. 207

The optimization run took more than 12 hours to run on a Windows machine and less than 1 hour 208

to run on a Unix machine. The 15 optimization runs required were distributed into 15 different 209

processors and therefore the entire computation required less than an hour to get the final optimal 210

solutions. If the same analysis was performed on a single Windows machine, it would have 211

taken more than 7 days to complete the optimization. Figure 6 describes the progress of the 212

Pareto-optimal front with the increasing generation number for sediment reduction. It is 213

observed that the optimization results provide a range of solutions for the pollution reduction 214

from the baseline. Similar analysis was performed for phosphorus and nitrogen. 215

Figure 7 presents the progress of the optimization front as the generation number increases for 216

pesticide control in LRW. It is observed that at the end of 5000 generations, the solutions are 217

spread well to provide a range of solutions for the selection and placement of BMPs in the 218

watershed for pesticide control. 8 different solutions spanning the entire spread of the Pareto-219

optimal front at the end of generations 10, 20, 50, 5000 are selected and evaluated using the 220

SWAT model to estimate the pollutant loads. These 32 different model runs were independent 221

11

of each other and therefore were evaluated in 32 different processors. This reduced the 222

computation time from 128 minutes on a single Windows machine to less than a minute when 223

run parallel on Steele. 224

Figure 9 presents the progress of the Pareto-optimal front as the generation number increases and 225

the Pareto-optimal front at the end of the optimization for P, N, and sediment reduction in LRW. 226

It is again observed that the Pareto-optimal front at the end of the final generation provides a 227

range of alternative solutions for selection and placement of BMPs that can be chosen based on 228

the required criteria for pollution reduction or available funds for BMP implementation. Figure 229

10 represents the SWAT simulated pollutant loads with the corresponding costs for the solutions 230

20 solutions picked from the range of the Pareto-optimal fronts (Figure 9) for generations 10, 40, 231

100, 1000, and 5000. The 20 picked solutions from the selected generations were set up to run 232

on 20 different processors. The overall computation time to run these 100 different solutions 233

was less than 5 minutes which would have been at least 16 hours when evaluated on a single 234

CPU running Windows. 235

236

Conclusion 237

It is observed from the study that high performance computing (HPC) helps reducing the overall 238

computation time to simulate multiple tasks by distributing them to run in parallel on multiple 239

processors. The overall computation time to run the optimization model for BMP selection and 240

placement was reduced by more than 90% by performing the runs in parallel on a supercomputer 241

in contrast to a single processor run on Windows. Also, the overall computation time required to 242

run the solutions picked from the simplified BMP tool based optimization technique to run the 243

12

SWAT model was also reduced by more than 90% because of the high performance 244

supercomputing alternative chosen when compared to single machine Windows run. The results 245

obtained from this study are encouraging to perform computationally intensive jobs on a 246

distributed parallel architecture. Future work in this regard could be to use high performance 247

computing develop a dynamically linked BMP selection and placement model that distributes the 248

various processes during a particular generation into multiple slave processors and the outputs 249

from the slave processors is received by the master processor that performs various optimization 250

routines to the variables based on the objective function values and therefore considerably 251

reducing the computation time. 252

253

254

255

13

REFERENCES 256

Arabi, M., R. S. Govindaraju, and M. M. Hantush (2006), Cost-effective allocation of watershed 257 management practices using a genetic algorithm, Water Resources Research, 42(10), 258 W10429. 259

260

Bekele, E. G., and J. W. Nicklow (2005), Multiobjective management of ecosystem services by 261 integrative watershed modeling and evolutionary algorithms, Water Resources Research, 262 41(10), 10406. 263

264

Confesor, R., and G. Whittaker (2007), Automatic calibration of hydrologic models with multi-265 objective evolutionary algorithm and Pareto optimization, Journal of the American Water 266 Resources Association, 43(4), 981-989. 267

268

Gitau, M. W., T. L. Veith, W. J. Gburek, and A. R. Jarret (2006), Watershed-level BMP 269 selection and placement in the Town Brook watershed, NY, Journal of the American 270 Water Resources Association, 42(6), 1565-1581. 271

272

Maringanti, C. (2008), Development of a multi-objective optimization tool for selection and 273 placement of BMPs for nonpoint source pollution control, M.S. Thesis. Purdue 274 University, 157. 275

276

Maringanti, C., I. Chaubey, and J. Popp (2009), Development of a Multi-Objective Optimization 277 Tool for the Selection and Placement of BMPs for Nonpoint Source Pollution Control, 278 Water Resources Research, (Accepted). 279

280

Srivastava, P., J. M. Hamlett, P. D. Robillard, and R. L. Day (2002), Watershed optimization of 281 best management practices using AnnAGNPS and a genetic algorithm, Water resources 282 research, 38(3), 1021. 283

284

Vrugt, J. A., B. Ó Nualláin, B. A. Robinson, W. Bouten, S. C. Dekker, and P. M. Sloot (2006), 285 Application of parallel computing to stochastic parameter estimation in environmental 286 models, Computers & Geosciences, 32(8), 1139-1155, doi:10.1016/j.cageo.2005.10.015. 287

288

289 290

14

291

Table 1. Computation time on Windows and Unix machines for various SWAT model 292 simulations used in the study 293

294

Watershed Years of simulation No. of HRUs Computation time

Windows Unix

L’Anguille River Watershed

18 433 10 mins 30 secs

Wildcat Creek Watershed

7 403 4 mins 15 secs

Lincoln Lake Watershed

28 1461 30 mins 3 mins

295 296

15

297 298

Figure 1. Pareto-optimum front progress during multi-objective optimization 299

300

Objective function 1 (0,0)

16

301

302

Figure 2. Land use distribution of Lincoln Lake Watershed (Source: Chiang et al, 2008). 303

304

305 306

17

307

308 Figure Error! No text of specified style in document.. Location of L’Anguille River Watershed 309 in Arkansas and the location of USGS stream gauging stations in the watershed 310

311

312 313

• Colt 07047942

• Palestine 07047950

18

` 314

Figure Error! No text of specified style in document.. Location of Wildcat Creek Watershed 315

in Indiana and the observed gauge locations 316

317 318

19

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

Figure 5. Flow chart that shows the methodology used in the study to distribute the 339 computationally intensive jobs (dashed arrows) to multiple processors. 340

341

SWAT BMP tool

Genetic Algorithm (NSGA-II)

BMP selection and placement tool

Nitrogen Phosphorus Sediment

5yr average pollutant load

10yr average pollutant load

15yr average pollutant load

20yr average pollutant load

25yr average pollutant load

Output Pareto-optimal front of solutions

Pick solutions

Run SWAT

Wildcat Creek and L’Anguille River watersheds

342

343

344

Figure 6. Change in the objective function as the number of generation increases for 5 different 345

annual averages considered for sediment346

15 year

the objective function as the number of generation increases for 5 different

annual averages considered for sediment in LLW.

25 year

5 year

5 year

20

the objective function as the number of generation increases for 5 different

20 year

10 year

21

347

348 Figure 7. Progress of the Pareto-optimal front during optimization of the model for pesticide control in LRW 349 350

22

351

Figure 8. Performance of the solutions obtained from the BMP tool when implemented at the 352

watershed level using the SWAT model to simulate the pesticide loads 353

354

0.4 0.5 0.6 0.7 0.8 0.9 1 1.1

0

20

40

60

80

100

120

140

↓Baseline

Avg. Pesticide Concentration (µg/L)

Net

Co

st (

$/h

a)

Generation 10Generation 20Generation 50Generation 5000

23

355

Figure 9. Pareto-optimal fronts during the optimization (Left, the numbers indicate the 356 generation number for the provided spread) and after the final generation (Right) for Sediment, 357

Phosphorus, and Nitrogen reduction 358

359

360

24

Figure 10. Simulation of the solutions obtained during optimization using the SWAT model for 361

estimating the (a) sediment, (b) phosphorus, and (c) nitrogen loads at Palestine gage station in 362

the watershed 363

364

b

c