[ op_ research @ im.ntu ]
DESCRIPTION
MANET Simulation Studies : The Incredibiles 2005 Mobile Computing and Communications Review, Volume 1, Number 2. Agenda:. Introduction Survey Results Common Simulation Pitfalls Simulation Setup Simulation Execution Output Analysis Publishing Community Resources Conclusion. - PowerPoint PPT PresentationTRANSCRIPT
[[OP_ ResearchOP_ Research @@ IM.NTU IM.NTU]]
Agenda:Agenda:
IntroductionIntroduction Survey ResultsSurvey Results Common Simulation PitfallsCommon Simulation Pitfalls
Simulation SetupSimulation Setup Simulation ExecutionSimulation Execution Output AnalysisOutput Analysis PublishingPublishing
Community ResourcesCommunity Resources ConclusionConclusion
MANET Simulation Studies : The IncredibilesMANET Simulation Studies : The Incredibiles2005 Mobile Computing and Communications Review, Volume 1, Number 22005 Mobile Computing and Communications Review, Volume 1, Number 2
指導教授:林永松2008/06/09
D95725009陳仁偉
OP@ IM. NTU 2008 Eric Chen .
[ 2 ]
AuthorAuthor
Education Ph.D. Computer Science, The College of William & Mary, 1993M.S. Computer Science, Michigan State University, 1989B.A. Mathematics, Kalamazoo College, 1987
Stuart Kurkowski
Michael Colagrosso
OP@ IM. NTU 2008 Eric Chen .
[ 3 ]
Mobile Ad Hoc Networks (MANETs) are wireless Mobile Ad Hoc Networks (MANETs) are wireless
mobile nodes that cooperatively form a network mobile nodes that cooperatively form a network
without Infrastructurewithout Infrastructure
Challenges:Challenges: Routing, wireless communications issues, Routing, wireless communications issues,
resource issues, resource issues, The leading is simulation
Simulation Study: ACM International Symposium on ACM International Symposium on
Mobile Ad Hoc Networking and Computing (MobiHoc) Mobile Ad Hoc Networking and Computing (MobiHoc)
from 2000-2005from 2000-2005
MANET simulation studies lack believability
Mobile Ad Hoc Networks (MANETs) are wireless Mobile Ad Hoc Networks (MANETs) are wireless
mobile nodes that cooperatively form a network mobile nodes that cooperatively form a network
without Infrastructurewithout Infrastructure
Challenges:Challenges: Routing, wireless communications issues, Routing, wireless communications issues,
resource issues, resource issues, The leading is simulation
Simulation Study: ACM International Symposium on ACM International Symposium on
Mobile Ad Hoc Networking and Computing (MobiHoc) Mobile Ad Hoc Networking and Computing (MobiHoc)
from 2000-2005from 2000-2005
MANET simulation studies lack believability
I. IntroductionI. Introduction
OP@ IM. NTU 2008 Eric Chen .
[ 4 ]
This study focused on the following four areas of credibility in research
1.1. RepeatableRepeatable
2.2. UnbiasedUnbiased
3.3. RigorousRigorous
4.4. Statistically soundStatistically sound
This study focused on the following four areas of credibility in research
1.1. RepeatableRepeatable
2.2. UnbiasedUnbiased
3.3. RigorousRigorous
4.4. Statistically soundStatistically sound
FocusFocus
OP@ IM. NTU 2008 Eric Chen .
[ 5 ]
MANET research MANET research full papers full papers published in published in MobiHocMobiHoc 114 out of the 151 MobiHoc papers published 114 out of the 151 MobiHoc papers published
(75.5%) used (75.5%) used simulationsimulation Unfortunately, 34 of the 114 published MobiHoc Unfortunately, 34 of the 114 published MobiHoc
simulation papers (29.8%) did not identify the simulation papers (29.8%) did not identify the
simulatorsimulator used in the research. used in the research. Network Simulator-2 (Network Simulator-2 (NS-2)NS-2) [34] is the most used [34] is the most used
simulator in MANET research; 35 of the 80 simulation simulator in MANET research; 35 of the 80 simulation
papers that state the simulator used in the papers that state the simulator used in the
simulation study used NS-2 (43.8%).simulation study used NS-2 (43.8%). No paper made a statement about No paper made a statement about code availability code availability
MANET research MANET research full papers full papers published in published in MobiHocMobiHoc 114 out of the 151 MobiHoc papers published 114 out of the 151 MobiHoc papers published
(75.5%) used (75.5%) used simulationsimulation Unfortunately, 34 of the 114 published MobiHoc Unfortunately, 34 of the 114 published MobiHoc
simulation papers (29.8%) did not identify the simulation papers (29.8%) did not identify the
simulatorsimulator used in the research. used in the research. Network Simulator-2 (Network Simulator-2 (NS-2)NS-2) [34] is the most used [34] is the most used
simulator in MANET research; 35 of the 80 simulation simulator in MANET research; 35 of the 80 simulation
papers that state the simulator used in the papers that state the simulator used in the
simulation study used NS-2 (43.8%).simulation study used NS-2 (43.8%). No paper made a statement about No paper made a statement about code availability code availability
I.A. The Current State of MANETI.A. The Current State of MANET Simulation Studies Simulation Studies
Repeatability - 1Repeatability - 1
OP@ IM. NTU 2008 Eric Chen .
[ 6 ]
OP@ IM. NTU 2008 Eric Chen .
[ 7 ]
The researcher must identify the The researcher must identify the simulatorsimulator and and
version,version, the the operating systemoperating system, and all , and all variable variable
settingssettings.. Repeatability is also based on the scenarios Repeatability is also based on the scenarios
evaluated, the techniques used to avoid evaluated, the techniques used to avoid initialization initialization
biasbias (influence of empty queues, etc., at the start), (influence of empty queues, etc., at the start),
and the techniques used to and the techniques used to analyze the resultsanalyze the results.. Thus, a Thus, a published paper published paper must discuss or reference all must discuss or reference all
of these details to meet the of these details to meet the repeatability criteriarepeatability criteria
The researcher must identify the The researcher must identify the simulatorsimulator and and
version,version, the the operating systemoperating system, and all , and all variable variable
settingssettings.. Repeatability is also based on the scenarios Repeatability is also based on the scenarios
evaluated, the techniques used to avoid evaluated, the techniques used to avoid initialization initialization
biasbias (influence of empty queues, etc., at the start), (influence of empty queues, etc., at the start),
and the techniques used to and the techniques used to analyze the resultsanalyze the results.. Thus, a Thus, a published paper published paper must discuss or reference all must discuss or reference all
of these details to meet the of these details to meet the repeatability criteriarepeatability criteria
I.A. The Current State of MANETI.A. The Current State of MANET Simulation Studies Simulation Studies
Repeatability -2Repeatability -2
OP@ IM. NTU 2008 Eric Chen .
[ 8 ]
Unbiased study: Unbiased study:
A project must address A project must address initialization biasinitialization bias, , random random
numbernumber issues, and issues, and use a variety of scenariosuse a variety of scenarios. .
Rigorous study:Rigorous study:
Factors such as Factors such as node densitynode density, , node footprintnode footprint, ,
coveragecoverage, , speedspeed, and , and transmission range transmission range must be must be
set to exercise the protocol under test. set to exercise the protocol under test.
Unbiased study: Unbiased study:
A project must address A project must address initialization biasinitialization bias, , random random
numbernumber issues, and issues, and use a variety of scenariosuse a variety of scenarios. .
Rigorous study:Rigorous study:
Factors such as Factors such as node densitynode density, , node footprintnode footprint, ,
coveragecoverage, , speedspeed, and , and transmission range transmission range must be must be
set to exercise the protocol under test. set to exercise the protocol under test.
I.A. The Current State of MANETI.A. The Current State of MANET Simulation StudiesSimulation Studies
OP@ IM. NTU 2008 Eric Chen .
[ 9 ]
Statistically sound study:Statistically sound study:
A project must account for A project must account for initialization biasinitialization bias, , executeexecute
a number of simulation iterations, provide the a number of simulation iterations, provide the
confidence levels confidence levels that exist in the results, and list that exist in the results, and list
any statistical assumptions made.any statistical assumptions made.
Statistically sound study:Statistically sound study:
A project must account for A project must account for initialization biasinitialization bias, , executeexecute
a number of simulation iterations, provide the a number of simulation iterations, provide the
confidence levels confidence levels that exist in the results, and list that exist in the results, and list
any statistical assumptions made.any statistical assumptions made.
I.A. The Current State of MANETI.A. The Current State of MANET Simulation StudiesSimulation Studies
OP@ IM. NTU 2008 Eric Chen .
[ 10 ]
Simulation type: - Simulation type: - terminating or steady-state
66 of the 114 simulation papers (57.9%) did not 66 of the 114 simulation papers (57.9%) did not
mention the type of simulation used in the study.mention the type of simulation used in the study.
Pseudo random number generator (PRNG)Pseudo random number generator (PRNG)
The goal is to raise awareness of the issues and to The goal is to raise awareness of the issues and to
introduce tools to aid MANET researchers in introduce tools to aid MANET researchers in
conducting and reporting credible simulation resultsconducting and reporting credible simulation results
Simulation type: - Simulation type: - terminating or steady-state
66 of the 114 simulation papers (57.9%) did not 66 of the 114 simulation papers (57.9%) did not
mention the type of simulation used in the study.mention the type of simulation used in the study.
Pseudo random number generator (PRNG)Pseudo random number generator (PRNG)
The goal is to raise awareness of the issues and to The goal is to raise awareness of the issues and to
introduce tools to aid MANET researchers in introduce tools to aid MANET researchers in
conducting and reporting credible simulation resultsconducting and reporting credible simulation results
I.B. Survey MotivationI.B. Survey Motivation
OP@ IM. NTU 2008 Eric Chen .
[ 11 ]
Simulation study questions.Simulation study questions.
Reviewed each paper individuallyReviewed each paper individually
For consistency, the same person reviewed all of the For consistency, the same person reviewed all of the
papers; to validate the results, we had a second papers; to validate the results, we had a second
person review all of the papers with a subset of the person review all of the papers with a subset of the
questions and a third person to correct the few questions and a third person to correct the few
inconsistencies found.inconsistencies found.
114 114 out of the 151 (75.5%) published MobiHoc out of the 151 (75.5%) published MobiHoc
papers used simulation as the basis for the study.papers used simulation as the basis for the study.
Simulation study questions.Simulation study questions.
Reviewed each paper individuallyReviewed each paper individually
For consistency, the same person reviewed all of the For consistency, the same person reviewed all of the
papers; to validate the results, we had a second papers; to validate the results, we had a second
person review all of the papers with a subset of the person review all of the papers with a subset of the
questions and a third person to correct the few questions and a third person to correct the few
inconsistencies found.inconsistencies found.
114 114 out of the 151 (75.5%) published MobiHoc out of the 151 (75.5%) published MobiHoc
papers used simulation as the basis for the study.papers used simulation as the basis for the study.
II. Survey ResultsII. Survey ResultsResearch DesignResearch Design
OP@ IM. NTU 2008 Eric Chen .
[ 12 ]
OP@ IM. NTU 2008 Eric Chen .
[ 13 ]
OP@ IM. NTU 2008 Eric Chen .
[ 14 ]
OP@ IM. NTU 2008 Eric Chen .
[ 15 ]
III.A. Simulation SetupIII.A. Simulation SetupIII.A.1. Simulation TypeIII.A.1. Simulation Type
III.A.2. Model Validation & VerificationIII.A.2. Model Validation & Verification
III.A.3. PRNG Validation & VerificationIII.A.3. PRNG Validation & Verification
III.A.4. Variable DefinitionIII.A.4. Variable Definition
III.A.5. Scenario DevelopmentIII.A.5. Scenario Development
III.B. Simulation ExecutionIII.B. Simulation ExecutionIII.B.1. Setting the PRNG SeedIII.B.1. Setting the PRNG Seed
III.B.2. Scenario InitializationIII.B.2. Scenario Initialization
III.B.3. Metric CollectionIII.B.3. Metric Collection
III.C. Output AnalysisIII.C. Output AnalysisIII.C.1. Single Set of DataIII.C.1. Single Set of Data
III.C.2. Statistical AnalysisIII.C.2. Statistical Analysis
III.C.3. Confidence IntervalsIII.C.3. Confidence Intervals
III.D. PublishingIII.D. Publishing
III.A. Simulation SetupIII.A. Simulation SetupIII.A.1. Simulation TypeIII.A.1. Simulation Type
III.A.2. Model Validation & VerificationIII.A.2. Model Validation & Verification
III.A.3. PRNG Validation & VerificationIII.A.3. PRNG Validation & Verification
III.A.4. Variable DefinitionIII.A.4. Variable Definition
III.A.5. Scenario DevelopmentIII.A.5. Scenario Development
III.B. Simulation ExecutionIII.B. Simulation ExecutionIII.B.1. Setting the PRNG SeedIII.B.1. Setting the PRNG Seed
III.B.2. Scenario InitializationIII.B.2. Scenario Initialization
III.B.3. Metric CollectionIII.B.3. Metric Collection
III.C. Output AnalysisIII.C. Output AnalysisIII.C.1. Single Set of DataIII.C.1. Single Set of Data
III.C.2. Statistical AnalysisIII.C.2. Statistical Analysis
III.C.3. Confidence IntervalsIII.C.3. Confidence Intervals
III.D. PublishingIII.D. Publishing
III. Common Simulation PitfallsIII. Common Simulation Pitfalls
OP@ IM. NTU 2008 Eric Chen .
[ 16 ]
III.A.1. Simulation Type - III.A.1. Simulation Type - terminating vs. steady-
state 66 out of the 114 simulation papers (57.9%) did 66 out of the 114 simulation papers (57.9%) did
not statenot state MANET : steady-state MANET : steady-state statistically unsound resultsstatistically unsound results
III.A.2. Model Validation & VerificationIII.A.2. Model Validation & Verification Simulation modelSimulation model The model must be validated as a baseline to The model must be validated as a baseline to
start any experimentationstart any experimentation Not validating the model or verifying code is a Not validating the model or verifying code is a
common pitfallcommon pitfall
III.A.1. Simulation Type - III.A.1. Simulation Type - terminating vs. steady-
state 66 out of the 114 simulation papers (57.9%) did 66 out of the 114 simulation papers (57.9%) did
not statenot state MANET : steady-state MANET : steady-state statistically unsound resultsstatistically unsound results
III.A.2. Model Validation & VerificationIII.A.2. Model Validation & Verification Simulation modelSimulation model The model must be validated as a baseline to The model must be validated as a baseline to
start any experimentationstart any experimentation Not validating the model or verifying code is a Not validating the model or verifying code is a
common pitfallcommon pitfall
III.A. Simulation SetupIII.A. Simulation Setup
OP@ IM. NTU 2008 Eric Chen .
[ 17 ]
III.A.3. PRNG Validation & VerificationIII.A.3. PRNG Validation & Verification NS-2 Cycle length is 2NS-2 Cycle length is 23131-1 -1 Akaroa-2 [11] offers an NS-2 compatible PRNG Akaroa-2 [11] offers an NS-2 compatible PRNG
with a cycle of 2with a cycle of 2191 191 − 1.− 1. 2-dimensional request on a PRNG is valid for 2-dimensional request on a PRNG is valid for
pproximately 8p3 L, where L is the cycle length.pproximately 8p3 L, where L is the cycle length. III.A.4. Variable DefinitionIII.A.4. Variable Definition
There are 538 variables defined in the ns-There are 538 variables defined in the ns-
default.tcl file of NS-2.1b7a and there are 674 default.tcl file of NS-2.1b7a and there are 674
variables defined in the ns-default.tcl file of NS-variables defined in the ns-default.tcl file of NS-
2.27.2.27. Our review of the Tcl driver files show that many Our review of the Tcl driver files show that many
simulation driver files leave key parameters simulation driver files leave key parameters
undefined.undefined.
III.A.3. PRNG Validation & VerificationIII.A.3. PRNG Validation & Verification NS-2 Cycle length is 2NS-2 Cycle length is 23131-1 -1 Akaroa-2 [11] offers an NS-2 compatible PRNG Akaroa-2 [11] offers an NS-2 compatible PRNG
with a cycle of 2with a cycle of 2191 191 − 1.− 1. 2-dimensional request on a PRNG is valid for 2-dimensional request on a PRNG is valid for
pproximately 8p3 L, where L is the cycle length.pproximately 8p3 L, where L is the cycle length. III.A.4. Variable DefinitionIII.A.4. Variable Definition
There are 538 variables defined in the ns-There are 538 variables defined in the ns-
default.tcl file of NS-2.1b7a and there are 674 default.tcl file of NS-2.1b7a and there are 674
variables defined in the ns-default.tcl file of NS-variables defined in the ns-default.tcl file of NS-
2.27.2.27. Our review of the Tcl driver files show that many Our review of the Tcl driver files show that many
simulation driver files leave key parameters simulation driver files leave key parameters
undefined.undefined.
III.A. Simulation SetupIII.A. Simulation Setup
OP@ IM. NTU 2008 Eric Chen .
[ 18 ]
III.A.5. Scenario DevelopmentIII.A.5. Scenario Development Only 48 of the 109 MANET protocol simulation Only 48 of the 109 MANET protocol simulation
papers in our survey of published MobiHoc papers papers in our survey of published MobiHoc papers
provided all three of these input parameters, provided all three of these input parameters,
detailing detailing 6161 simulation scenarios. simulation scenarios. #36 & #37#36 & #37 number of nodes : 10 – 30000 Simulation area : 25m x 25m - 5000m x 5000 m. Transmission ranges: 3m - 1061 m.Transmission ranges: 3m - 1061 m. The survey results highlight the wide range of The survey results highlight the wide range of
simulation scenarios used to conduct MANET simulation scenarios used to conduct MANET
research and the lack of uniform rigorous testing research and the lack of uniform rigorous testing
of MANET protocols.of MANET protocols.
III.A.5. Scenario DevelopmentIII.A.5. Scenario Development Only 48 of the 109 MANET protocol simulation Only 48 of the 109 MANET protocol simulation
papers in our survey of published MobiHoc papers papers in our survey of published MobiHoc papers
provided all three of these input parameters, provided all three of these input parameters,
detailing detailing 6161 simulation scenarios. simulation scenarios. #36 & #37#36 & #37 number of nodes : 10 – 30000 Simulation area : 25m x 25m - 5000m x 5000 m. Transmission ranges: 3m - 1061 m.Transmission ranges: 3m - 1061 m. The survey results highlight the wide range of The survey results highlight the wide range of
simulation scenarios used to conduct MANET simulation scenarios used to conduct MANET
research and the lack of uniform rigorous testing research and the lack of uniform rigorous testing
of MANET protocols.of MANET protocols.
III.A. Simulation SetupIII.A. Simulation Setup
OP@ IM. NTU 2008 Eric Chen .
[ 19 ]
OP@ IM. NTU 2008 Eric Chen .
[ 20 ]
III.A.5. Scenario DevelopmentIII.A.5. Scenario Development We validated the wide range of input parameters We validated the wide range of input parameters
by comparing the by comparing the derived parameters derived parameters of each of each
scenario.scenario. Figure 2 is a scatter plot of all the derived Figure 2 is a scatter plot of all the derived
parameters for the 61 sets of input parameters.parameters for the 61 sets of input parameters. The scatter plot reflects the wide range of The scatter plot reflects the wide range of
scenarios and the scenarios and the lack of correlation lack of correlation between between
parameters. parameters. Figure 2 also shows the Figure 2 also shows the lack of independence lack of independence
between parameters, such as between parameters, such as node density node density and and
node coverage node coverage .. the lack of the lack of multiple groupings multiple groupings in each plot in each plot
illustrates that the community is not covering the illustrates that the community is not covering the
range of values in a consistent organized range of values in a consistent organized
manner.manner.
III.A.5. Scenario DevelopmentIII.A.5. Scenario Development We validated the wide range of input parameters We validated the wide range of input parameters
by comparing the by comparing the derived parameters derived parameters of each of each
scenario.scenario. Figure 2 is a scatter plot of all the derived Figure 2 is a scatter plot of all the derived
parameters for the 61 sets of input parameters.parameters for the 61 sets of input parameters. The scatter plot reflects the wide range of The scatter plot reflects the wide range of
scenarios and the scenarios and the lack of correlation lack of correlation between between
parameters. parameters. Figure 2 also shows the Figure 2 also shows the lack of independence lack of independence
between parameters, such as between parameters, such as node density node density and and
node coverage node coverage .. the lack of the lack of multiple groupings multiple groupings in each plot in each plot
illustrates that the community is not covering the illustrates that the community is not covering the
range of values in a consistent organized range of values in a consistent organized
manner.manner.
III.A. Simulation SetupIII.A. Simulation Setup
OP@ IM. NTU 2008 Eric Chen .
[ 21 ]
OP@ IM. NTU 2008 Eric Chen .
[ 22 ]
OP@ IM. NTU 2008 Eric Chen .
[ 23 ]
III.A.5. Scenario DevelopmentIII.A.5. Scenario Development The MANET community needs a way to The MANET community needs a way to
characterize characterize simulation scenarios simulation scenarios in order to in order to
evaluate and compare protocols and performance evaluate and compare protocols and performance
and ensure protocols are rigorously tested.and ensure protocols are rigorously tested. For example, from Table 2, scenario #8, the For example, from Table 2, scenario #8, the
simulation area is 3000mx 3000 m, but the simulation area is 3000mx 3000 m, but the
transmission range of 1061m lowers the average transmission range of 1061m lowers the average
hop count to only 1.67 hops. This hop count hop count to only 1.67 hops. This hop count
means most source and destination pairs are means most source and destination pairs are
direct neighbors and the rest have only one direct neighbors and the rest have only one
intermediate node.intermediate node.
III.A.5. Scenario DevelopmentIII.A.5. Scenario Development The MANET community needs a way to The MANET community needs a way to
characterize characterize simulation scenarios simulation scenarios in order to in order to
evaluate and compare protocols and performance evaluate and compare protocols and performance
and ensure protocols are rigorously tested.and ensure protocols are rigorously tested. For example, from Table 2, scenario #8, the For example, from Table 2, scenario #8, the
simulation area is 3000mx 3000 m, but the simulation area is 3000mx 3000 m, but the
transmission range of 1061m lowers the average transmission range of 1061m lowers the average
hop count to only 1.67 hops. This hop count hop count to only 1.67 hops. This hop count
means most source and destination pairs are means most source and destination pairs are
direct neighbors and the rest have only one direct neighbors and the rest have only one
intermediate node.intermediate node.
III.A. Simulation SetupIII.A. Simulation Setup
OP@ IM. NTU 2008 Eric Chen .
[ 24 ]
III.B.1. Setting the PRNG SeedIII.B.1. Setting the PRNG Seed NS-2 uses a default seed of 12345 for each NS-2 uses a default seed of 12345 for each
simulation runsimulation run Produce identical resultsProduce identical results Negate the independent replication methodNegate the independent replication method None of the 84 simulation papers addressed None of the 84 simulation papers addressed
PRNG issues.PRNG issues.
III.B.1. Setting the PRNG SeedIII.B.1. Setting the PRNG Seed NS-2 uses a default seed of 12345 for each NS-2 uses a default seed of 12345 for each
simulation runsimulation run Produce identical resultsProduce identical results Negate the independent replication methodNegate the independent replication method None of the 84 simulation papers addressed None of the 84 simulation papers addressed
PRNG issues.PRNG issues.
III.B. Simulation ExecutionIII.B. Simulation Execution
OP@ IM. NTU 2008 Eric Chen .
[ 25 ]
III.B.2. Scenario InitializationIII.B.2. Scenario Initialization This pitfall usually occurs from a lack of This pitfall usually occurs from a lack of
understanding of the two types of simulation. understanding of the two types of simulation.
terminating or steady-state Only Only eight eight of the 114 simulation papers in our of the 114 simulation papers in our
MobiHoc survey (7.0%) addressed initialization MobiHoc survey (7.0%) addressed initialization
bias, and bias, and all eight use the unreliable method of all eight use the unreliable method of
arbitrarily deleting dataarbitrarily deleting data..
III.B.2. Scenario InitializationIII.B.2. Scenario Initialization This pitfall usually occurs from a lack of This pitfall usually occurs from a lack of
understanding of the two types of simulation. understanding of the two types of simulation.
terminating or steady-state Only Only eight eight of the 114 simulation papers in our of the 114 simulation papers in our
MobiHoc survey (7.0%) addressed initialization MobiHoc survey (7.0%) addressed initialization
bias, and bias, and all eight use the unreliable method of all eight use the unreliable method of
arbitrarily deleting dataarbitrarily deleting data..
III.B. Simulation ExecutionIII.B. Simulation Execution
OP@ IM. NTU 2008 Eric Chen .
[ 26 ]
III.C.1. Single Set of DataIII.C.1. Single Set of Data This pitfall is taking the This pitfall is taking the first set of results first set of results from a from a
simulation and accepting the results as “truth”.simulation and accepting the results as “truth”. Only 39 of the 109 MANET protocol simulation Only 39 of the 109 MANET protocol simulation
papers (35.8%) stated the papers (35.8%) stated the number of simulation number of simulation
runs executed.runs executed.
III.C.2. Statistical AnalysisIII.C.2. Statistical Analysis This pitfall concerns not using the This pitfall concerns not using the correct correct
statistical formulasstatistical formulas with the different forms of with the different forms of
output.output.
III.C.1. Single Set of DataIII.C.1. Single Set of Data This pitfall is taking the This pitfall is taking the first set of results first set of results from a from a
simulation and accepting the results as “truth”.simulation and accepting the results as “truth”. Only 39 of the 109 MANET protocol simulation Only 39 of the 109 MANET protocol simulation
papers (35.8%) stated the papers (35.8%) stated the number of simulation number of simulation
runs executed.runs executed.
III.C.2. Statistical AnalysisIII.C.2. Statistical Analysis This pitfall concerns not using the This pitfall concerns not using the correct correct
statistical formulasstatistical formulas with the different forms of with the different forms of
output.output.
III.C. Output AnalysisIII.C. Output Analysis
OP@ IM. NTU 2008 Eric Chen .
[ 27 ]
III.C.3. Confidence IntervalsIII.C.3. Confidence Intervals Confidence intervals are a tool to provide a range Confidence intervals are a tool to provide a range
where we think the population mean is located where we think the population mean is located
relative to the point estimaterelative to the point estimate
9898 of the 112 simulation papers using plots of the 112 simulation papers using plots
(87.5%) did not show confidence intervals on the (87.5%) did not show confidence intervals on the
plots.plots.
III.C.3. Confidence IntervalsIII.C.3. Confidence Intervals Confidence intervals are a tool to provide a range Confidence intervals are a tool to provide a range
where we think the population mean is located where we think the population mean is located
relative to the point estimaterelative to the point estimate
9898 of the 112 simulation papers using plots of the 112 simulation papers using plots
(87.5%) did not show confidence intervals on the (87.5%) did not show confidence intervals on the
plots.plots.
III.C. Output AnalysisIII.C. Output Analysis
OP@ IM. NTU 2008 Eric Chen .
[ 28 ]
The inconsistency prevents the direct comparison of The inconsistency prevents the direct comparison of
results, limiting research advancements.results, limiting research advancements. A new researcher cannot repeat the studies to start A new researcher cannot repeat the studies to start
his or her own follow-on research.his or her own follow-on research. When publishing, the authors need to state if the When publishing, the authors need to state if the
code is available and how to obtain the codecode is available and how to obtain the code..
The inconsistency prevents the direct comparison of The inconsistency prevents the direct comparison of
results, limiting research advancements.results, limiting research advancements. A new researcher cannot repeat the studies to start A new researcher cannot repeat the studies to start
his or her own follow-on research.his or her own follow-on research. When publishing, the authors need to state if the When publishing, the authors need to state if the
code is available and how to obtain the codecode is available and how to obtain the code..
IV.D. PublishingIV.D. Publishing
OP@ IM. NTU 2008 Eric Chen .
[ 29 ]
The The Akaroa-2Akaroa-2 suite, which help a researcher suite, which help a researcher
monitor simulation execution to determine steady-monitor simulation execution to determine steady-
state and prevent correlation among multiple state and prevent correlation among multiple
replications of a simulation.replications of a simulation. The interactive NS-2 protocol and environment The interactive NS-2 protocol and environment
confirmation tool (confirmation tool (iNSpectiNSpect) , which ) , which visualizesvisualizes the the
trace file of an NS-2 simulation. The visualizations trace file of an NS-2 simulation. The visualizations
can be used for scenario development, model can be used for scenario development, model
validation, protocol verification, and results analysis.validation, protocol verification, and results analysis. The Simulator for Wireless Ad Hoc Networks The Simulator for Wireless Ad Hoc Networks
(SWAN) , which enables a researcher to create a (SWAN) , which enables a researcher to create a
virtual environment for conducting experiments with virtual environment for conducting experiments with
MANETs.MANETs.
The The Akaroa-2Akaroa-2 suite, which help a researcher suite, which help a researcher
monitor simulation execution to determine steady-monitor simulation execution to determine steady-
state and prevent correlation among multiple state and prevent correlation among multiple
replications of a simulation.replications of a simulation. The interactive NS-2 protocol and environment The interactive NS-2 protocol and environment
confirmation tool (confirmation tool (iNSpectiNSpect) , which ) , which visualizesvisualizes the the
trace file of an NS-2 simulation. The visualizations trace file of an NS-2 simulation. The visualizations
can be used for scenario development, model can be used for scenario development, model
validation, protocol verification, and results analysis.validation, protocol verification, and results analysis. The Simulator for Wireless Ad Hoc Networks The Simulator for Wireless Ad Hoc Networks
(SWAN) , which enables a researcher to create a (SWAN) , which enables a researcher to create a
virtual environment for conducting experiments with virtual environment for conducting experiments with
MANETs.MANETs.
IV. Community ResourcesIV. Community Resources
OP@ IM. NTU 2008 Eric Chen .
[ 30 ]
Summarizing the four areas of credibility, we found Summarizing the four areas of credibility, we found
less than less than 15%15% of the published MobiHoc papers are of the published MobiHoc papers are
repeatable.repeatable... Only seven of the 58 MobiHoc simulation papers that Only seven of the 58 MobiHoc simulation papers that
use a public simulator (12.1%) mention the simulator use a public simulator (12.1%) mention the simulator
versionversion used. used. Only Only eight eight of the 114 simulation papers (7.0%) of the 114 simulation papers (7.0%)
addressed addressed initialization bias initialization bias and none of the 84 and none of the 84
simulation papers addressed random number simulation papers addressed random number
generator issues.generator issues. statistical soundness, statistical soundness, 70 70 of the 109 MANET protocol of the 109 MANET protocol
simulations papers (64.2%) did not identify the simulations papers (64.2%) did not identify the
number of simulation iterations number of simulation iterations used, and 98 of the used, and 98 of the
112 papers that used plots to present simulation 112 papers that used plots to present simulation
results (87.5%) did not include results (87.5%) did not include confidence intervals.confidence intervals.
Summarizing the four areas of credibility, we found Summarizing the four areas of credibility, we found
less than less than 15%15% of the published MobiHoc papers are of the published MobiHoc papers are
repeatable.repeatable... Only seven of the 58 MobiHoc simulation papers that Only seven of the 58 MobiHoc simulation papers that
use a public simulator (12.1%) mention the simulator use a public simulator (12.1%) mention the simulator
versionversion used. used. Only Only eight eight of the 114 simulation papers (7.0%) of the 114 simulation papers (7.0%)
addressed addressed initialization bias initialization bias and none of the 84 and none of the 84
simulation papers addressed random number simulation papers addressed random number
generator issues.generator issues. statistical soundness, statistical soundness, 70 70 of the 109 MANET protocol of the 109 MANET protocol
simulations papers (64.2%) did not identify the simulations papers (64.2%) did not identify the
number of simulation iterations number of simulation iterations used, and 98 of the used, and 98 of the
112 papers that used plots to present simulation 112 papers that used plots to present simulation
results (87.5%) did not include results (87.5%) did not include confidence intervals.confidence intervals.
V. ConclusionsV. Conclusions
OP@ IM. NTU 2008 Eric Chen .
[ 31 ]
Hence, only approximately Hence, only approximately 12%12% of the MobiHoc of the MobiHoc
simulation results appear to be based on simulation results appear to be based on sound sound
statistical statistical techniques.techniques. This survey of MobiHoc papers showed the current This survey of MobiHoc papers showed the current
state of MANET research and the state of MANET research and the lack of consistencylack of consistency, ,
re-enforcing the need for simulation study guidance.re-enforcing the need for simulation study guidance. The goals is to make the research repeatable, The goals is to make the research repeatable,
unbiased, rigorous, and statistically soundunbiased, rigorous, and statistically sound
Hence, only approximately Hence, only approximately 12%12% of the MobiHoc of the MobiHoc
simulation results appear to be based on simulation results appear to be based on sound sound
statistical statistical techniques.techniques. This survey of MobiHoc papers showed the current This survey of MobiHoc papers showed the current
state of MANET research and the state of MANET research and the lack of consistencylack of consistency, ,
re-enforcing the need for simulation study guidance.re-enforcing the need for simulation study guidance. The goals is to make the research repeatable, The goals is to make the research repeatable,
unbiased, rigorous, and statistically soundunbiased, rigorous, and statistically sound
V. ConclusionsV. Conclusions