The Delphi Method

1

Use of the Delphi Method to Prioritize Events Impacting Operations in the Maritime Domain

Victor Zaloom, Ph.D. & Vinit Subhedar

Lamar University, Texas, USA

The Delphi Method

2

ABSTRACT

The objective of this research is to demonstrate that the Delphi technique can be used to

gather expert opinion to identify and prioritize terrorist events, natural disasters, and failures

(equipment and people) in the maritime domain. A Delphi study was conducted to identify

events that could disrupt shipping and to rank their likelihood of occurrence on the Sabine-

Neches Waterway, a strategic maritime artery located in south east Texas. Participants in the

study included experts from five sectors of the maritime domain: public ports, United States

Coast Guard (USCG), shipping industry, private ports, and law enforcement. The study team that

managed the process included an engineering professor who is engaged in studying maritime

risks and a master of engineering student. Results indicate that a panel of experts were able to

achieve a high level of agreement regarding the types of events causing risks to shipping and

their relative likelihood

Key Words: Delphi Method, Maritime Risk

The Delphi Method

3

INTRODUCTION

Terrorism and natural disasters have had a significant impact on our national priorities.

This fact is especially true since terrorist attacks on the World Trade Center, and the Pentagon,

and hurricanes Katrina [2005], Rita [2005], Wilma [2005], Gustav [2008], and Ike [2008]. Huge

investments have been made, and more will be made to reduce risks to our population,

infrastructure, and way of life. Risk must be properly assessed in order to get the best returns on

these investments.

In the next section, Maritime Risk Factors and Security Risk, the importance of

identifying and prioritizing events causing risks is explained. The section that follows, Delphi

Method, discusses the application of the Delphi technique to identify and prioritize events

causing risks in the maritime domain. Then implementation of the Delphi research study is

described as a ten step process including selection of a panel of experts. Survey results are

analyzed using the standard deviations to assess the level of convergence of the panel of experts.

Conclusions from the study are presented in the final section of the report.

The Delphi Method

4

MARITIME RISK FACTORS AND SECURITY RISK

World trade is important to economic growth of the U.S. and global economies. The

maritime domain is vital for world trade and commerce. Its efficient supply chain including

super tankers and containerized shipping and seamless inter-modal connections are key driving

factors accelerating the flow of world commerce. Inter-modal connections refer to places in the

transportation system where the transportation mode changes such as ship to rail car or rail car to

truck. Consumer, commercial, and military goods are moved efficiently with great ease with the

help of advanced and efficient transportation modes. To secure the maritime domain is

challenging because of its inter-modal connections, long distances, and global nature. The

maritime domain’s safety and efficiency are a global responsibility further shared by

international, national, and local maritime organizations (International Maritime

Organization (IMO) 2006, Department of Homeland Security (DHS) 2004).

The maritime domain’s logistic chain for most international commerce begins at the

manufacturer and/or shipper in the country of origin and ends with the buyer in the destination

country. It includes customs at both the origin and destination countries, port operations in both

countries, and the maritime carrier. Stakeholders’ involvement from shipper to buyer and

customs, port operators of both origin and destination countries are responsible for the domain’s

safety and security. The United States uses a layered approach to security in the maritime

domain. Many security initiatives have been implemented as shown in the Figure 1. The

Container Security Initiative (CSI) and the Customs-Trade Partnership against Terrorism (C-

TPAT) focus on container security. They are the two principal American voluntary programs

(Barnes, 2004).

The Delphi Method

5

The potential threat in the maritime domain includes containers, cargoes, people, ships,

vessels, port facility infrastructure, and, last but not the least, natural hazards. Barnes states, up to

90% of world cargo movement occurs in shipping containers with up to 250 million movements

each year and a mere 2% of this volume is physically inspected post-arrival (2004, Organization

for Economic Co-operation and Development 2003).

Figure 1. Maritime Domain’s Logistic Chain and Scope of IMO and U.S. Security Initiatives

Source: (Organization for Economic Co-operation and Development 2003)

The September 11, 2001, attack on the World Trade Center and the Pentagon attracted

much-needed attention to the subject of security risk. Different tools and methodologies can be

used for security risk assessment. Security related risk can be assessed by identifying internal

and external threat scenarios linked with a particular event and negative consequences caused by

that event.

DESTINATION COUNTRY

MANUFACTURER/

SHIPPER

CUSTOMS

PORT

OPERATOR

MANUFACTURER /

SHIPPER

CUSTOMS

Buying Agent

Transport Operator

( land / inland

waterway)

Freight Forwarder

Consolidator

NVOCC

Bank

Customhouse

Brokers

De-consolidator

Transport Operator

(land / inland

waterway)

Bank

IMO ISPS

US CSI

US C-TPAT

MARITIME

CARRIER

COUNTRY OF ORIGIN

PORT

OPERATOR

BUYER

The Delphi Method

6

A recent paper (Mitchell and Decker 2004) entitled:“Applying Risk-based Decision-

making Methods/Tools to U.S. Navy Antiterrorism Capabilities” highlights the importance of

understanding the elements of “security risk” and its assessment. Security Risk associated with

an event can be expressed with the help of the traditional risk equation,

Risk = Probability (F) x Consequence (C) (EQ. 1)

In the case of security risks as opposed to risks from natural disasters the probability can be

separated into two parts: threat and vulnerability and equation EQ. 1 is modified as,

Security Risk = [Threat (T) x Vulnerability (V)] x Consequence (C) (EQ. 2)

Threat can be perceived as a possessed risk in a situation. A situation can have multiple

threat scenarios, and each scenario is very useful in the risk assessment stage. “Vulnerability” is

all about susceptibility or weakness. Each threat scenario will have different levels of

vulnerabilities depending on the safeguards or resilience provided for that particular situation or

scenario. What is important is that vulnerability for each threat scenario should be analyzed. The

third component of security risk is “Consequence”; it is the magnitude of the negative impact if

an unwanted event occurs. Total risk is the sum of all risks taken over all possible events.

The security improvement process can be explained with the help of Figure 2. The steps

are (1) identify risk, (2) prioritize risk, (3) allocate resources, (4) design risk mitigation

strategies, and (5) implement risk mitigation strategies.

There are many techniques and/or tools available to accomplish each of these steps. Risk

identification can result from a brainstorming meeting where experts from several domains or

different sectors within a domain are asked to identify risks associated with a specific property or

populations. Another technique for risk identification is to do a global search for certain types of

events and to determine which ones could happen in the domain of interest.

The Delphi Method

7

Figure 2. Security Improvement Process

After risks are identified, they must be prioritized by considering factors: the probability

of occurrence of an “event” and expected damage resulting if that event occurs. The probability

of occurrence depends on a number of considerations including vulnerability and threat.

Vulnerability is a function of the degree of protection against the event. Threat is a function of

the attractiveness of the target. Blocks 1 and 2 of Figure 2 form the risk assessment stage and are

the focus of this research. Events causing risks are identified and prioritized with the use of

qualitative and/or quantitative methods. The third step is to allocate resources to reduce or

eliminate various threats or to reduce their associated vulnerability. Proper allocation of

resources holds the key for a successful security improvement process.

The fourth step is to design mitigation strategies. One example of mitigation strategy is to

increase the level of alertness. This might involve increasing the frequency of surveillance,

tightening the security on access to property or populations. Another strategy would be to guard

2. Prioritize Risks

3. Allocate

Resources

5. Implement Mitigation

Strategies

1. Identify Risks

4. Design Risk

Mitigation

Strategies

Risk Assessment

The Delphi Method

8

a population or property. An example of a guard is to build a sea wall to prevent or mitigate

flooding from a hurricane. Another example would be to lock and guard the cockpit cabin of a

commercial airplane. One more strategy might be to install warning systems such as sirens,

flashing lights, labels, or public announcements. Risk mitigation techniques must be effectively

implemented to protect populations and/or property. In this study we are concentrating on the

risk associated with the maritime domain and concerned with the “events” that could disrupt

shipping on the Sabine – Neches Waterway for a period of twenty-four hours or more.

The Delphi Method, a structured process of collecting and using the qualitative

knowledge from a group of experts by means of a series of questionnaires was used to

accomplish blocks 1 and 2 of Figure 2. Subjective estimates from the experts were gathered,

analyzed, and used as suitable information for deciding what events are most likely to cause

security risks.

DELPHI METHOD

The Delphi concept was first initiated in early 1950’s by the California-based firm

RAND Corporation. The concept was used in defense research and was named Project Delphi.

The objective of the study was to “obtain the most reliable consensus of opinion of a group of

experts by a series of intensive questionnaires interspersed with controlled opinion feedback”

(Dalkey and Helmer 1963). The application of the Delphi Method was very specific in the field

of forecasting. The intention was to gather experts’ opinion into a consensus or a statement that

can be used as valid input in generating future scenarios. The justification of the Delphi’s

application in 1960’s for forecasting was based on the limitations in technological resources to

develop quantitative models or to conduct extensive data collection and analysis. The technique

The Delphi Method

9

has been used in various fields such as environmental, health, and sales forecasting (Linstone and

Turoff, 1975).

The method incorporates two important elements. The first element is anonymity among

group members. This is required to encourage diverse opinions based on the expertise of each

individual. The second element is controlled feedback in which the responses from the group

members are gathered and summarized and then fed back to the group. Each panel member is

asked to consider his/her responses in relation to the responses of the rest of the group. Based on

this, they are asked to respond to the next round of the survey. The method can be viewed as a

“controlled debate”. Extreme opinions from the group members are made open and clear via the

controlled feedback, and estimates are achieved bypassing the problems of group dynamics

(Gordon 1994). In our study the rankings of each member of the panel of experts are presented

to every panel member without identifying who provided these rankings.

The study discussed below was of interest to the research team because the team leader

served on the National Maritime Security Advisory Committee (NMSAC) that is charged by the

Secretary of Homeland Security to advise the United States Coast Guard on matters of maritime

security.

DELPHI STUDY

Results of following the ten step procedure shown in Figure 3 are reported in this section.

The first step was to form a research team. Our research team was led by an engineering

professor who was a member on the inaugural National Maritime Security Advisory Committee

(NMSAC). This committee, mandated by the United States Congress, advises the Commandant

of the USCG on matters of maritime security. The other member of the study team was a Master

of Engineering Science student who selected this study for his masters’ thesis topic.

The Delphi Method

10

1. Formation of a team to undertake and monitor a Delphi on a given subject.

2. Selection of one or more panels to participate in the exercise.

Customarily, the panelists are experts in the area to be investigated.

3. Development of the first round Delphi questionnaire.

4. Testing the questionnaire for proper wording (e.g. ambiguities).

5. Transmission of the first questionnaire to the panelists.

6. Analysis of the first round responses.

7. Preparation of the second round questionnaire (and possible testing).

8. Transmission of the second round questionnaire to the panelists.

9. Analysis of the second round responses (Steps 7 to 9 are reiterated as long as

desired or necessary to achieve stability in the results).

10. Preparation of a report by the analysis team to present the conclusions of the

exercise.

Figure 3. Ten Step Delphi Study Procedure; Fowles (1979)

The second step involved the selection of a panel of experts on maritime security. This

group defined and prioritized events causing maritime security risks. By law the United Stated

Coast Guard (USCG) controls shipping into and out of all U.S. ports. Therefore, they are

knowledgeable of all sectors of the maritime domain associated with ports and waterways. So the

research team leader met with representatives of the USCG to select sectors to be represented on

the panel and specific individuals from those sectors. The panel of experts used, to accomplish

The Delphi Method

11

the project objective represented five core areas of the maritime domain. These areas, shown in

Figure 4, are public ports, the U.S. Coast Guard, the shipping industry, private ports/docks, and

law enforcement.

Figure 4. The Delphi Project Panel of Experts

A survey instrument (See Figure 5.) to use in the first round of the survey of the panel of

experts was developed in the third step of the Delphi procedure. This instrument was developed

by considering the various types of events causing maritime risks as shown in Figure 6. The

original survey consisted of ten events and space where participants could suggest additional

events. After reviewing this instrument with survey experts not on the panel (Step 5) it was

distributed by email to participants along with a request that they add additional events if needed

or suggest modifications to existing events. Additional information provided to survey

participants included historical data on past major hurricanes and natural disasters and detailed

instructions on ranking the likelihood of events. Results of the first round of the survey (Table 1)

were received by email or fax and analyzed (Step 6). The average, lowest, and highest priority

ranking for each event was tabulated and provided to survey participants with instructions for

Experts from the

Maritime Domain

Public Ports

U.S.C.G.

Shipping

Industry

Private Ports/

Docks

Law

Enforcement

Delphi Project

on the

Maritime

Domain

Affiliation

Experts’

Knowledge

The Delphi Method

12

Sr.No.

List of “Events”

Likelihood Scale

1 (most likely) to

10 (least likely)

1

Major vessel accident causing waterway closure or

disruption of vessel traffic for more than one hour.

(Unintentional, Human Error)

2

Major oil spill/leakage affecting the Sabine – Neches

Waterway. (Unintentional)

3

Port facility infrastructure breakdown. (Unintentional)

4

Damage to port facility, from a vessel equipment

failure/malfunction. (Unintentional, Navigational)

5

Terror threat (hoax) – causing shutdown of port

facilities or parts of the Sabine – Neches Waterways.

6

Damage or destroy a large vessel or tanker with the help

of a small vessel approaching it with an explosive on

board. (Terrorist Acts, USS Cole-type Act)

7 Disruption of port facilities and/or operations by

destroying key assets or infrastructure such as cranes,

electrical power systems, etc.(Intentional, Terrorist

Acts)

8

Introduction into the United States of a weapon of mass

destruction via the Sabine – Neches Waterway.

(Intentional, Terrorist Acts)

9 Coastal storms and hurricanes up to category II (Natural

Disaster)

10 Category III, IV, or V hurricanes. (Natural Disaster)

Suggest Additional “Events” if you choose

A

B

Figure 5. Delphi Survey Round 1 Form

The Delphi Method

13

Figure 6. Maritime Risk Categories

round 2 (Step 7). In survey round 1 (SR1), event 1, major vessel accident causing waterway

closure or disruption of vessel traffic for more than one hour (unintentional, human error) was

the most likely with an average ranking of 1.2 with highest ranking of 1 and lowest ranking of 2.

Event 2, major oil spill/leakage affecting the Sabine – Neches Waterway (unintentional), scored

an average 2.3 and was the second most likely event on the likelihood scale. Its highest rank was

1, and its lowest rank was 5. Event 9, coastal storms and hurricanes up to category II (natural

disaster), scored an average of 3.3 and was ranked the third most likely event with highest rank

Events Causing Maritime Risks

Natural Disasters Man–made Events

Tropical Storm,

Hurricane

Severe

Weather

Intentional

Unintentional

(Errors/Failures)

Fog

Tornado

Terrorism

Sabotage

Human Errors

Procedural

Lapses

Failures

Mechanical

Breakdown

Structured

The Delphi Method

14

Table 1. Survey Round 1 (SR1) Summary Data

Panel

Members

Group 1 Group 2 Group 3 Group 4 Group 5 Survey Round 1 (SR1)

Analysis

Public Ports

U.S. Coast

Guard

Shipping Industry

Private Ports

Law Enforcem

ent

Quantitative tabulation of responses from participants

Responses for the

Rankings

SP1

SP2

SP1

SP2

SP1

SP2

SP1

SP2

SP1

SP2

Avg.

Low

High

Rank

Event 1

1

2

1

1

1

1

1

2

1.2

2

1

1

Event 2

5

1

2

2

4

2

2

1

2.3

5

1

2

Event 3

6

8

3

6

6

6

4

5

5.5

8

3

6

Event 4

4

5

4

7

5

3

3

4

4.3

7

3

4

Event 5

7

9

5

5

7

5

6

6

6.2

9

5

7

Event 6

8

6

9

9

9

9

7

9

8.2

9

6

9

Event 7

9

7

8

8

8

7

9

8

8.0

9

7

8

Event 8

10

10

10

10

10

10

10

10

10.0

10

10

10

Event 9

2

3

6

2

2

4

5

3

3.3

6

2

3

Event 10

3

4

7

4

3

8

8

7

5.5

8

3

5

SP1, SP2 = Survey Participant 1, 2 SP1, SP2 = Survey Participant 1 from Group 1 and Survey

Participant 2 from Group 5 opted out prior to completing Survey Round 1

of 2 and lowest 6. This event was rated high by group 3 (shipping industry) participants. Event 8,

introduction into the United States of a weapon of mass destruction via the Sabine – Neches

Waterway (intentional, terrorist acts) was ranked lowest by every participant and was the least

likely event on the list.

The Delphi Method

15

Two suggestions were received for additional events to our original list of ten events that

might cause disruption of vessel traffic on the Sabine - Neches Waterway. One suggestion

referred to dense fog was received from one of the participants from the USCG. Therefore, dense

fog was explicitly added to the reworded event 9 on the survey round 2. Event 9 was changed to

read: “Dense fog, coastal storms, hurricanes up to category II and other similar weather

conditions.” This change was suggested by one of the participants from the shipping industry

group, and he gave the highest likelihood ranking to event 9. Another suggestion was to include

an event caused by underwater improvised explosive devices. It was suggested by a participant

from the private port, group 4. Based on the suggestion, this type of event was added to the

reworded event 6. It was reworded for survey round 2 as follows: “Damage or destroy a large

vessel or tanker with the help of a small vessel approaching it with explosives on board or by

underwater improvised explosive devices” (terrorist acts, USS Cole-type act). Another

modification to the survey form for round 2 is related to event 1. This event referred to

“disruption of vessel traffic for more than one hour.” The reworded event 1 reads: “Disruption of

vessel traffic for more than 24 hours.” This change was made because it was determined that

short term delays would not materially affect the flow of commerce on the Sabine – Neches

Waterway.

The survey round 2 contained the reworded events 1,6, and 9 and the average, lowest,

and highest ranking for each event from survey round 1. This completed the reiteration of Step 7.

The revised survey form was emailed to participants (Step 8 reiteration). They were asked to

review the summary data from round 1 and the reworded events and reconsider their rankings.

The results of survey round 2 appear in Table 2. The average, lowest, highest, and overall

ranking of each event was obtained and provided to survey participants.

The Delphi Method

16

Table 2. Survey Round 2 (SR2) Summary Data

Panel

Members

Group 1 Group 2 Group 3 Group 4 Group 5 Survey Round 2 (SR2)

Analysis

Public Ports

U.S. Coast

Guard

Shipping Industry

Private Ports

Law Enforcem

ent

Quantitative tabulation of responses from participants

Responses for the

Rankings

SP1

SP2

SP1

SP2

SP1

SP2

SP1

SP2

SP1

SP2

Avg.

Low

High

Rank

Event 1

1

3

2

2

1

1

1

2

1.6

3

1

1

Event 2

5

2

3

3

4

2

2

1

2.7

5

1

3

Event 3

6

7

4

5

5

6

5

5

5.3

7

4

5

Event 4

4

4

5

4

3

3

3

4

3.7

5

3

4

Event 5

7

9

6

6

7

5

6

6

6.5

9

5

7

Event 6

8

6

9

8

9

9

8

9

8.2

9

6

8

Event 7

9

8

8

9

8

7

9

8

8.2

9

7

9

Event 8

10

10

10

10

10

10

10

10

10.0

10

10

10

Event 9

2

1

1

1

2

4

4

3

2.2

4

1

2

Event 10

3

5

7

7

6

8

7

7

6.2

8

3

6

SP1, SP2 = Survey Participant 1, 2 SP1, SP2 = Survey Participant 1 from Group 1 and Survey

Participant 2 from Group 5 opted out prior to completing Survey Round 1.

The Delphi Method

17

ANALYSIS OF DELPHI SURVEY RESULTS

Standard deviation of the likelihood rankings for each event of the panel of experts was

used to measure level of agreement or convergence. Standard deviations of responses for each

event were compared on survey rounds 1 (SR1) and 2 (SR2) as shown in Figure 7. In all cases

except events 1 and 8, the standard deviations of the responses (rankings) decreased from survey

round 1 to survey round 2. The definition of event 1 was significantly changed from survey

round 1 to survey round 2, i.e., in survey round 2 the event called for a closure of the waterway

for twenty-four hours versus only one hour in survey round 1. This change in event 1 definition

resulted in three survey participants decreasing their ranking of this event while five participants

kept their ranking the same. Consequently variation (standard deviation) of event 1 rankings

increased from round 1 to round 2. Event 8 had no variation (standard deviation) on either round

1 or round 2. All participants agreed it was the least likely event to cause closure of the

waterway.

The fact that standard deviations of responses decreased or remained the same from

survey round 1 to survey round 2 for all events except for event 1, which had a change in

definition, indicates the panel members rankings were converging. The next most converged

events after event 8 were events 7 and 4 with standard deviations of 0.7 each. Events 1 and 3

were the next events that the group agreed upon with the standard deviations of 0.74 and 0.91

respectively. Five of ten events scored standard deviations of group responses for each event

between the range from 0 – 1.0. Events 6 and 5 had standard deviations of 1.03 and 1.19

respectively. Events 2 and 9 each had standard deviation of 1.28.

The Delphi Method

18

Figure 7. Standard Deviations of Responses for Each Event on Survey Rounds 1 and 2 (SR1 and

SR2)

Four out of the ten events held the same ranking in survey round 2 as in survey round 1.

Event 3 (unintentional port facility infrastructure breakdown) increased in overall likelihood

ranking from number 6 to number 5. Event 6 (USS Cole type of terrorists act) increased in

overall likelihood ranking from number 9 to number 8, and event 9 (Coastal storms) increased

from number 3 to number 2. Event 2 (major oil spill) decreased in overall ranking from number 2

to number 3, event 7 (intentional port facility infrastructure breakdown) decreased from number

8 to number 9, and event 10 (Category III and higher hurricanes) decreased from number 5 to

number 6 from survey round 1 (SR1) to survey round 2 (SR2).

0.46

1.4

1.51

1.3

1.38

1.16

0.75

0

1.5

2.2

0.74

1.28

0.91

0.7

1.19

1.03

0.7

0

1.28

1.58

0

0.25

0.5

0.75

1

1.25

1.5

1.75

2

2.25

2.5

Event 1 Event 2 Event 3 Event 4 Event 5 Event 6 Event 7 Event 8 Event 9 Event 10

Events to occur on the Sabine - Neches Waterway

St. Dev. of Value (Rank)

SR1St. Dev SR2 St. Dev

The Delphi Method

19

Table 3. Rankings of Each Event after Each Survey Round

Survey Results

Events Round 1

Rank

Round 2

Rank

Event 1(major vessel accident,E1) 1 1

Event 2(major oil spill,E2) 2 3

Event 3(facility damage due to navigational error) 6 5

Event 4(facility damage due to equipment failure) 4 4

Event 5(terror threat-hoax) 7 7

Event 6(USS Cole type terrorist act) 9 8

Event 7(facility damage due to terrorism) 8 9

Event 8(intro. of weapons of mass destruction) 10 10

Event 9(coastal storms) 3 2

Event 10(category III or above hurricane) 5 6

The Delphi Method

20

CONCLUSIONS

• The Delphi method is a useful technique to use to assess the likelihood of events causing

risks in the maritime domain. Delphi panel members (maritime domain security experts)

were comfortable in ranking events that could disrupt operations on the Sabine – Neches

Waterway on the likelihood scale from the most likely to least likely to occur in a

calendar year.

• The analysis of rounds 1 and 2 questionnaires suggest that a consensus was reached on

identifying and prioritizing the events that could cause the closure of the Sabine – Neches

Waterway for twenty-four hours or more.

• Events from the man-made events: category, unintentional including human errors and

failures and natural disasters category scored high on the likelihood scale. The least likely

events were from the intentional - terrorist acts category.

• The findings of Delphi research indicated that subjective judgment on a collective basis

can be used to provide multidisciplinary expertise for a comprehensive security risk

assessment of the maritime domain.

• Since natural disasters, accidents, and failures are ranked so much higher than terrorist

acts on the likelihood and the probability scale, efforts and funds expended on detailed

recovery plans should reflect the relative likelihood.

REFERENCES

The Delphi Method

21

Barnes, P. 2004. Security risk management issues in maritime trade: An analysis. Academy of

International Business (AIB) Southeast Asia Regional Conference. 2004.

http://eprints.qut.edu.au/archive/00001162/01/4_Barnes.pdf (accessed September 23,

2005).

Dalkey, N. and O. Helmer. 1963. An experimental application of the Delphi Method to the use of

experts. Management Science 9 (3): 458-467.

Department of Homeland Security. 2004. Secure Seas, Open Ports.

http://supplychainsecurity.biz/documents/DHSPortSecurityFactSheet-062104.pdf

(accessed February 26, 2007)

Fowles, J. 1978. The Delphi technique: Handbook of future research. London: Greenwood

Place.

Gordon, T. 1994. The Delphi Method in futures research methodology, AC/UNC Millennium

Project. http://www.futurovenezuela.org/_curso/5-delphi.pdf (accessed April 15, 2006).

International Maritime Organization 2006. http://www.imo.org ,

http://www.imo.org/includes/blastDataOnly.asp/data_id%3D16406/5.pdf

(accessed December 17, 2006).

Linstone, A. and M. Turoff (eds). 1975. The Delphi Method: Techniques and applications.

Massachusetts: Addison-Wesley. Also available online at http://is.njit.edu/turoff

Mitchell, C. and C. Decker. 2004. Applying risk-based decision-making methods/tools to U.S.

Navy antiterrorism capabilities. Journal of Homeland Security.

http://www.homelanddefense.org/journal/Articles/Mitchell_Decker.html (accessed

October 28, 2006).

The Delphi Method

22

Murphy, M., N. Black, D. Lamping, C. McKee, C.F.B. Sanderson, J. Aksham, and T. Marteau.

1998. Consensus development methods and their use in clinical guideline development.

Health Technology Assessment 2(3).

http://www.hta.nhsweb.nhs.uk/fullmono/mon203.pdf (accessed April 21, 2006).

Organization for Economic Co-operation and Development (OECD). 2003. Security in maritime

transport: Risk factors and economic impact, Maritime Transport Committee, Directorate

for Science, Technology and Industry.

http://www.oecd.org/dataoecd/63/13/4375896.pdf (accessed October 7, 2005).

Woo, G. 2005. A Terrorism Risk Analyst’s Perspective on TRIA. Risk Management Solutions.

http://www.rms.com/Publications/RiskAnalystPersectiveTRIA_WooForCongress.pdf

(accessed October 7, 2005)