Journal of Conventional Weapons Destruction Journal of Conventional Weapons Destruction

Volume 10 Issue 1 The Journal of Mine Action Article 44

August 2006

Test and Evaluation of Japanese GPR-based AP Mine Detection Test and Evaluation of Japanese GPR-based AP Mine Detection

Systems Mounted on Robotic Vehicles Systems Mounted on Robotic Vehicles

Jun Ishikawa Japan Science and Technology Agency

Mitsuru Kiyota Japan Science and Technology Agency

Katsuhisa Furuta Tokyo Denki University

Follow this and additional works at: https://commons.lib.jmu.edu/cisr-journal

Part of the Defense and Security Studies Commons, Emergency and Disaster Management Commons,

Other Public Affairs, Public Policy and Public Administration Commons, and the Peace and Conflict

Studies Commons

Recommended Citation Recommended Citation Ishikawa, Jun; Kiyota, Mitsuru; and Furuta, Katsuhisa (2006) "Test and Evaluation of Japanese GPR-based AP Mine Detection Systems Mounted on Robotic Vehicles," Journal of Mine Action : Vol. 10 : Iss. 1 , Article 44. Available at: https://commons.lib.jmu.edu/cisr-journal/vol10/iss1/44

This Article is brought to you for free and open access by the Center for International Stabilization and Recovery at JMU Scholarly Commons. It has been accepted for inclusion in Journal of Conventional Weapons Destruction by an authorized editor of JMU Scholarly Commons. For more information, please contact dc_admin@jmu.edu.

92| research and development |journalofmineaction|2006|august|10.1 10.1|august|2006|journalofmineaction| research and development | 93

A t current clearance speed, it will take more than 100 yearsto remove all the landmines that remain in the world.1Consequently,Japanisdevelopingmoreefficientandsaferhu-

manitariandeminingtechnologies.ThisarticleintroducesJapaneserobotic sensor systems thatprovidedeminerswith clear subsurfaceimagesviaground-penetratingradarincombinationwithmetalde-tectors(GPR+MD).

Experiment Overview: BackgroundToreconstructclearimages,highlyaccuratesensor-positioning

systems,aswellassensingtechnologyitself,are indispensablebe-

of Japanese GPR-based AP Mine Detection Systems Mounted on Robotic Vehicles

by Jun Ishikawa and Mitsuru Kiyota [ Japan Science and Technology Agency ] and Katsuhisa Furuta [ Tokyo Denki University ]

This article introduces Japanese activities re-

garding a project, “Research and Development

of Sensing Technology, Access and Control

Technology to Support Humanitarian Demining

of AP Mines.” This project, which includes the

research of six teams from academia and in-

dustry, has been funded by the Japan Science

and Technology Agency (JST) under the auspices

of the Ministry of Education, Culture, Sports,

Science and Technology (MEXT ). The devel-

oped systems are equipped with both ground-

penetrating radar and a metal detector, and they

are designed to make no explicit alarm and to

leave decision-making of detection using subsur-

face images to the operators. To evaluate these

kinds of systems, a series of trials was conducted

in Japan from 8 February to 11 March 2005.

Test and Evaluation

cause one of themost important pieces of information for signalprocessingissensorposition,wherethesensoracquiresaseriesofdataforGPR+MD.

Therearemanykindsofanti-personnellandmines,whichcanbelaidbyhumansor scatteredby airplanes, andmined areas arenotlimitedtoplainsbutalsomarshes,canals,steephillsides,seashores,deserts,mountainsandforests.Forsuchroughterrain,roboticsys-temsmusthavesensorheadsthatcanscanthegroundascloselyaspossiblebutnevertouchitaswell-traineddeminersdo.Metaldetec-tors,whichareakindofanelectromagneticinduction(EMI)sensor,havethepossibledetectiondistanceofabout1�centimetersformini-mum-metallandmines.Forthesemetaldetectors,itisachallengeforsensorsystemstoaccessminefieldsandmanipulatethesensorheadinsevereenvironmentsinordertostayasclosetothegroundaspossible.Thus,Japaneseadvancedroboticsandsensorengineeringhavebeenfusedtocreatenoveldetectors.

Japan started preparation for this kind of research and develop-ment inMarch1997,whentheTokyoConferenceonAnti-personnelLandmineswasheld.Atthisconference,participantsundertookacom-prehensivediscussiontostrengtheninternationaleffortstowardaddress-ingtheproblemsofAPlandmines,especiallylandmineclearancebytheUnitedNationsandotherorganizations;developmentofnewtechnologyforminedetectionandremoval;andassistancetovictims.InDecember1997,KeizoObuchi,thenMinisterforForeignAffairsofJapan,signedtheOttawaConvention,2andtheultimategoalofzerovictimswaspro-posed.SinceAugust2002,theJapanesehaveundertakenpreparationstostarthumanitarian-deminingR&D.3

Japanese R&D of Anti-personnel Landmine Detection System

WithstrongexpectationsfromtheworldcommunityforJapanesecontributionsinthisarea,theMinistryofEducation,Culture,Sports,Science and Technology established the Committee of Experts onHumanitarianDeminingTechnologyinJanuary2002,believingintheimportanceoftacklingthetechnologicaldevelopmentofAPland-minedetectionusingadvancedJapanesetechnology.TheCommittee’sfindingswerepresentedtoMEXTinthereport,“PromotingR&DforHumanitarianDeminingTechnology.”�Basedonthisreport,theJapanScienceandTechnologyAgencyannouncedacallforproposalsforR&Dprojectsinhumanitarian-deminingtechnology.Outofthe�2proposals,12projectswereselected,andanR&Dprojectnamed“Research and Development of Sensing Technology, Access andControl Technology to Support Humanitarian Demining of Anti-personnelMines”startedinOctober2002.

The JST project is essentially divided into a short-term R&Dprojectandamedium-termone.Becauseoftheurgentneedforthistechnology,theshort-termR&Dproject isexpectedtohaveproto-typesinfieldtrialswithinthreeyears.TheJSTmedium-termR&Dprojectisonafive-yearschedule.Thegoalistodevelopsensingtech-nologiesthatcandetecttheexplosiveitself,intherangeofabout30to100grams.

RESEARCH &DEVELOPMENTdesign research development test innovation prevention design research development test innovation prevention design research development test innovation prevention design research development test innovation prevention design research development test innovation prevention design research development test innovation prevention design research development test innovation prevention design research development test innovation prevention design research development test innovation prevention design research development test innovation prevention design research development test innovation prevention design research development test innova-tion prevention design research development test innovation prevention design research development test innovation prevention design research development test innovation prevention

desig

n r

esearch d

evelo

pm

ent

test

innovation p

revention d

esig

n

research d

evelo

pm

ent

test

innovation p

revention d

esig

n r

esearch

develo

pm

ent

test

innovation p

revention d

esig

n r

esearch d

evelo

p-

ment

test

innovation p

revention

desig

n r

esearch d

evelo

pm

ent

test

innovation p

revention d

esig

n

research d

evelo

pm

ent

test

innovation p

revention d

esig

n r

esearch

develo

pm

ent

test

innovation p

revention d

esig

n r

esearch d

evelo

p-

ment

test

innovation p

revention

desig

n r

esearch d

evelo

pm

ent

test

innovation p

revention d

esig

n

research d

evelo

pm

ent

test

innovation p

revention d

esig

n r

esearch d

evelo

pm

ent

test

innova-

tion p

revention d

esig

n r

esearch d

evelo

pm

ent

test

innovation p

re-

vention d

esig

n r

esearch d

evelo

pm

ent

test

innovation p

revention

desig

n r

esearch d

evelo

pm

ent

test

innovation p

revention d

esig

n

research d

evelo

pm

ent

test

innovation p

revention

1

Ishikawa et al.: Test and Evaluation of Japanese GPR-based AP Mine Detection Systems Mounted on Robotic Vehicles

Published by JMU Scholarly Commons, 2006

9�| research and development |journalofmineaction|2006|august|10.1 10.1|august|2006|journalofmineaction| research and development | 9�

wave detector, such as an induction coil, detects subsequent NQRsignalsfromthe1�Nifanyintendedtargetexists,andtheresonancefrequency of the signal is unique for each explosive material. Thusexplosivescanbeidentified.

Tworesearchteamsontheprojectaretryingtodevelopdetectorsbased on the neutron analysis identifying explosives through back-scattering of neutrons and detection of specific energy gamma rays

fromcaptureonhydrogenandnitrogenatomsofexplosives.ProfessorKiyoshiYoshikawa’sgroupfromKyotoUniversityhasprototypedanextremelycompactneutronsourcebasedonan inertial-electrostaticconfinement fusion device 20 centimeters in diameter.1� ProfessorTetsuoIguchi’sgroupofNagoyaUniversityhasprototypedanotherneutron source, which is an improved Cockcroft-Walton-type ac-celerator neutron source using a deuterium-deuterium (DD) fusionreaction.Theyhavealsodevelopedaprototypeofamulti-Comptongammacamera,whichestimatestheincomingdirectionof10.�MeVgamma-raysproducedfromthenitrogenoftheexplosive(Figure2).16

Themedium-termR&Dprojectisexpectedtohaveprototypesinfieldtrialswithinfiveyears,namelyin2007,incombinationwithoneoftheprototypesofMHV,AMSorGryphon.

Experimental Design17,18

Toevaluatetheshort-termR&Dprototypes,aseriesoftestswasconductedfrom�Februaryto11March200�inSakaideCity,Japan.Seventestlaneswereconstructedusingmorethan200landminesur-rogates(Figure�).Sinceoperators’pre-knowledgeofthelocationsofburiedtargetssignificantly influencesthedetectionresults forsuchsystemsthatmakenoexplicitalarm,lanes1to6aredesignedtobeusedforblindtests.

Testlanesandlandminesurrogates.Inconstructingtestlanes,alltheoriginalsoilwasremovedfromawidthof2meterstoadepthof0.�metersintheverticalsection,andthelaneswerefilledwithho-mogeneousandnon-mineralized(cooperative)soil.Theactualwidthof test lanes is1meter,andminesurrogateswereburiedshallowerthanorequaltoadepthof0.3meters(1foot).Thefeaturesofeachlaneareasfollows:

• Lanes1,2and3are1�meterslongwithaflatsurface.• Lane�is20meterslong,with1�bumpsinthesurface,eachwith

aheightof10centimetersandadiameterof60centimeters,andsmallstonesaremixedtomakethesoilheterogeneous.

• Lane�simulatesminefieldsinpost-clearanceinspectionaftermechanicaldemining,withthesoilstirredandnotpacked.

• Lane6iswet,with10litersofwaterpersquaremetersprinkledonehourbeforetheteststarts.

Figure�showsfourkindsoflandminesurrogatesusedinthetest.TheM1�19andPMN220containametalpart—an1�-millimeter21verticalcarbonsteelpinwithadiameterof3millimeters—andtheType7222hasa�-millimeterverticalcarbonsteelpinwithadiam-eterof�millimeters.TheType72-S23mineismadebymodifyingaproductofAmtechAeronauticalLimited2�andhasexactlythesamemetalpartastheInternationalTestOperationsProcedurestandardI

0, a 12.7-millimeter vertical aluminum tube. Silicone rubber was

substitutedforexplosivesinallthesurrogates.Experimentaldesign.Throughthetests,influencesofvariousfactors

onprobabilityofdetectionshouldbeevaluated.Namely,inExperiment1,targettypes,targetdepth,soilconditionsandtargetangleswerecho-

senasfactorstobetested.TherearetwoorfourlevelsforeachfactorasdescribedinTable1.Accordingtothe soil conditions, for example,targets (landmine surrogates) thatare classified into “flat,” “wet,”“stirred” and “rough” are respec-tivelyburiedinlanes1–3,6,�and�ataspecifieddepthandangleasdefinedinFigure�.Experiment2was designed to mainly evaluate

Figure 3: Test-lane layout and the calibration area.

Figure 2: Multi-Compton gamma camera based on stacked BGO scintillator rods.

Figure 4: Landmine surrogates used in test.

Short-term R&D project. The objectives ofthe short-term R&D project are to develop sensingtechnology thatcansafelyandefficientlydetectAPlandminesbasedonthephysicaldifferencesbetweenlandmines and soils, and to develop access devicesandmanipulationtechnologythatcarrysensorsintominefields and allow them to scan the ground pre-cisely.Morespecifically,thegoalistodevelopvehicle-mounted GPR+MD dual-sensor systems that makeno explicit alarm and provide operators with clearsubsurface images.Thismeans that thedecision todeterminewhetherornotashadowintheimageisarealAPlandmineisentirelylefttotheoperator,simi-lartohowmedicaldoctorscanfindcancerbyreadingCT images. This feature discriminates the systemsfrom conventional GPR+MD dual sensors that arebasedonalarmtones.

Intheshort-termproject,foursensorsandthreero-boticvehicleshavebeendeveloped.OneofthoseistheMineHunterVehicle.ThevehicleitselfandthemanipulatorhavebeendevelopedbyaresearchteamofProfessorKenzoNonami’satChibaUniversity.�TheMHVcaninterchangeablymounttwoGPRsensorsinadditiontoacommercial,off-the-shelfmetaldetector.

One sensor is a stepped-frequencyGPRdevelopedbyProfessorMotoyukiSato’steamatTohokuUniversity,6hereinafterreferredtoasMHV#1.Stepped-frequencyradardeterminesdistancetoatargetbyconstructingasyntheticrangeprofile,whichisatimedomainap-proximationderivedfromthefrequencyresponseofacombinationofstepped-frequencysignalsviainversefastFouriertransform(IFFT).The major advantage of the stepped-frequency method is that thespectrumbandwidthcanbeeasilytunedtofitanoptimumvalueac-cordingtoenvironmentalconditionssuchassoilmoisture.

TheothersensorisanimpulseGPRdevelopedbyProfessorIkuoArai’sprojectat theUniversityofElectro-Communications,7herein-afterreferredtoasMHV#2.ThiskindofGPRoperatesbytransmit-tingaverynarrowpulse(<1nanosecond)ofelectromagneticwave,theadvantageofwhichisthatthemeasurementtimerequiredtogenerateonerangeprofileisveryshort.AftertheGPRscanstoacquirearangeprofileforeveryintervalofseveralcentimeters,�GPRtomographygivessubsurfacehorizontalslicesasshowninFigure1a,andfurthercalcula-tionprovidesoperatorswiththree-dimensionalimages(Figure1b).

ProfessorToshioFukuda’sgroupatNagoyaUniversitydevel-oped a dual sensor with built-in stepped-frequency GPR+MD.9

Thesensorsystemscanstheground,beingcarriedbyalow-reaction-force manipulation frame that has four balloons on the legs tosoftly landitonminefields.ThemanipulationframeisattachedtothetopofaboomofacranevehicledevelopedbyMr.TomohiroIkegami’s group at TADANO Ltd. The vehicle has a 20-meterreach for a 200-kilogram payload with a positioning accuracyof1� centimeters.These elementshavebeen integrated into theAdvancedMineSweeper(AMS),whichcanadapttovariousgeo-graphicalenvironments.10

Professor Shigeo Hirose’s team at the Tokyo Institute ofTechnologydevelopedtheGryphonbuggysystem,whichcanbere-motelycontrolledtoaccessminefields.11Themanipulatormountedonthebuggyhasbeendesignedtocancelreactionforceinducedbysensorscanning.12ThesensorisaGPR+MDdualsensornamedtheAdvancedLandmineImagingSystem(ALIS),anditcanalsobeusedasahandhelddetector.13ALISwasdevelopedbyProfessorSato’steamandunderwentafieldtrialinAfghanistaninDecember200�.

Medium-termR&Dproject.ProfessorHideoItozaki’sgroupofOsakaUniversityisdevelopinganuclearquadrupole-resonancedetec-tor.1�Intheanalysis,aradio-frequencyelectromagneticwaveisfirstemittedandexcitesnuclearspinof1�Ninexplosives.Thenamagnetic

All

figu

res

cour

tesy

of

Japa

n Sc

ienc

e an

d T

echn

olog

y A

genc

y.

Figure 1a: Detection images from stepped-frequency GPR. Horizontal slices showing two targets at a five-centimeter depth (left) and a target at a 25-centimeter depth (right).

Figure 1b: Detection images from stepped-frequency GPR. Three-dimensional image of three targets in the horizontal slices.

2

Journal of Conventional Weapons Destruction, Vol. 10, Iss. 1 [2006], Art. 44

https://commons.lib.jmu.edu/cisr-journal/vol10/iss1/44

96| research and development |journalofmineaction|2006|august|10.1 10.1|august|2006|journalofmineaction| research and development | 97

Table 3: Design result for Experiment 1.

Table 4: Design result for Experiment 2.

Table 5: Definition of confidence rating.

Table 6: PD of eight testees of Experiment 1. Highlighted data of four testees are analyzed as shown in Figure 13.

theminimumdiscriminationdistance.Twolevelswerechosenforthefactor“distancetoadjacenttarget”asdescribedinTable2.Onelevelconsistsofpairsoftargetsinadistanceof1�centimetersandtheotherlevelconsistsofindependenttargets,theseparationofwhichshallbeatleast�0centimeters.

Duetothelimitationoftimeforthetrialandthenumberoftargets,itisimpossibletotestallthecombinationsoflevelsinTables1and2.Toimpartiallycollectunbiaseddataforstatisticalanalysisunderthislimitation,orthogonalexperimentaldesignsbasedonL

16(21�)and

L�(27)orthogonalarrayswererespectivelyusedforExperiments1and

2.Assigningthecolumnsof thearraytoeachfactoras specifiedinTables1and2derivesareducedsetofcombinations,theresultsofwhicharesummarizedinTables3and�.Forexample,thenumberofexperimentalrunscanbereducedfrom12�(�×�×�×2)to16inExperiment1.

AccordingtoTables3and�(seeoppositepage),all the targetswereburiedatrandomlocationsinthespecifiedlanesandwereleftformorethanonemonthbeforethetestbegan.Testeescansubmitalltheimpartialdataneededforstatisticalanalysisbyreportingdetec-tionresultsfromlanes1through6.Inthetrial,atleasttwotesteesfromeverydevicetookthetestinall6lanes.

Benchmarking.TocompareperformanceoftheGPR+EMIdualsensorswith thatof existingmetaldetectors, abenchmarking trialwasconducted.Namely,atesterwhoknewtheexactpositionsoftar-getscheckedifanymetal-detectorresponseoccurredjustaboveeveryburiedtarget.Theresultofthistestshowsthebestperformanceofthemetaldetectorsused.

Testprocedures.Testeestookblindtestsforeachlanefollowingtheproceduresasdescribedbelow:

1. Before the test starts, the tester records temperature, relativehumidity and volumetric water content that is measured bytimedomainreflectometry(TDR).2�

2. Thetesteedoesclose-indetectionworkusingasensorsystemcooperativelywithvehicleoperators.

3. Aftertheworkfinishes,thetesterrecordstemperature,relativehumidityandvolumetricwatercontentmeasuredbyTDR.

�. Thetesteereportsthefollowingdataforeverydetectedanomaly: • Coordinatesofthedetectedtarget • Depthofthedetectedtarget

• ConfidenceratingdefinedinTable�andthefinaldeci-sion whether or not to declare the anomaly as a land-minesurrogate

�. The tester determines whether the declared anomaly can beconsideredtobefromtheintendedtargets,26thatis,withinadetectionhalo,theradiusofwhichishalfofthetargetdiam-eterplus10centimeters.27

6. Finally,thetesterclassifiesthereporteddataintofourcategories:• Truepositive:Thetesteedeclareditasatargetandthis

istrue.• Falsepositive:Thetesteedeclareditasatargetandthisis

nottrue.Thisisafalsealarm.• Truenegative:Thetesteedeclareditasafragment,clutter

ornoiseandthisistrue.• Falsenegative:Thetesteedeclareditasafragment,clutter

ornoiseandthisisnottrue.Thisismissingatarget.Completingthetestsfromlanes1through6meansthatthetestee

finishedall2�experimentalrunsofExperiments1and2describedinTables3and�.

Themostimportantthingistopracticallyusethesetechnologiestoimprovelandmine-detectionefficiencyandreduceminefields.Todoso,themine-detectionsystemsmustberobust,simpleandhighlycost-effective.TheJapanesedomestictrialisthefirststep.

Test and Evaluation ResultsThefollowingisthedataanalysisandevaluationoftestresultsfor

anti-personnellandminedetectionsystemsusingground-penetratingradar mounted on robotic vehicles for humanitarian demining.17,1�

The test results showed that combiningGPRwithmetaldetectorscanimproveprobabilityofdetectionfortargetsaroundadepthof20centimeters,where it isdifficulttodetect thetargetsbyusingonlya metal detector. It has also been learned that positioning controlmustbeimprovedinscanningthegroundwithasensorhead,whichiskeytomakingthebestofuseofmetaldetectorsmountedonve-hicles.Lessonslearnedhavebeenreflectedinfurtherimprovementoftheprototypes.Inthefollowingsections,dataanalysis,methodsandevaluationresultsaredescribed.

Data analysis.According to the experimental designproposedabove,datafromeighttestees(twoeachfromeverysystem)havebeenacquired.Thecomprehensiveresultsofprobabilityofdetection(PD)areshowninTables6and7andwereacquiredthroughExperiments1and2.ThesystemsnamedareanonymousanddescribedasDevice

Figure 5: Definitions of target depth and angle.

Table 2: Factors A to C and the levels for Experiment 2.

Table 1: Factors A to D and the levels for Experiment 1.

Continued on page 98, TEST

3

Ishikawa et al.: Test and Evaluation of Japanese GPR-based AP Mine Detection Systems Mounted on Robotic Vehicles

Published by JMU Scholarly Commons, 2006

9�| research and development |journalofmineaction|2006|august|10.1 10.1|august|2006|journalofmineaction| research and development | 99

Now,forexample,alinearmodelfortheprobabilityofdetectionp1canbedefinedas:

FortheANOVA,fourmeansofsquares(variances)arecalculatedasfollows:

whereƒC and

ƒe arethedegreesoffreedomoffactorsanderror.

Bycomparingthevariancesduetolevelsofeachfactor(i.e.,VA,VB

andVCwiththevarianceduetomeasurementerror(Ve)

usingF-test),

thesignificanceofthedifferencesbetweenlevelsistested.Inthistest,thenullhypothesisisthatthemaineffectsoflevelsforafactorareallequal(i.e.,thereisnodifferenceininfluencesoflevelsforthefactoronPD).ThecomputedFstatisticinTable9followsanFdistributionwith corresponding degrees of freedomunder the assumption thatvariancesofPDhavehomogeneity.29Therefore, the significanceofFcanbedeterminedintheusualwaybyusingthetableofF.IfthecomputedvalueofFislargerthanthetabledvalue,thenullhypoth-esis is rejected.Thismeans thatat leastonepairofmaineffects issignificantlydifferent.

The 9�-percent confidence limit of each main effect is experi-mentallyderivedbyusing

Ve,themeanofsquaresduetoerror.For

example,the9�-percentconfidenceintervalofa1�cmisgivenby:

wherend=�isthetotalnumberofexperiments(thenumberofex-perimentalrunsmultipliedbyrepetitions),andtƒe,95% isthequan-tileofthet-distributionforprobability9�percentwith

ƒe degrees

offreedom.Receiver operating characteristic curve. It has been30 years

sinceradiographicapplicationsofROCcurveswerereported30anditiswell-knownthatanalysisbasedonROCcurvesissuitableforsub-jectiveevaluationofimagingequipment.Inthetestandevaluationhere,ROCcurveswerealsousedtoevaluatesensoreffectivenessintermsofbothPDandfalse-alarmrate.

Table 9: Analysis of variance (ANOVA).

Asdescribedabove,detectionresultsreportedbytesteesareclas-sifiedintofourcategories:truepositive,falsepositive,truenegativeandfalsenegative.However,theclassificationbasedonatestee’sdis-crimination threshold is a one-sided view, and thenumberof truepositivesandthenumberof falsepositiveschangeas the thresholdisvaried.AnROCcurveshowsustherelationshipbetweenthetruepositiveandfalsepositiveforavarietyofdifferentthresholds,thushelpingthedeterminationofanoptimalthresholdaswellasthecom-parisonofsensorperformance.

ToplotanROCcurve,twohistograms,whicharemeasuredonanintervalscaleintheconfidenceratingreportedbythetestee,areneed-ed.Oneisfromsignalsofintendedtargetsthatconsistoftruepositivesandfalsenegatives,andtheotherisfromsignalsoffragments,clut-tersornoise(i.e.,truenegativesandfalsepositives).Accordingtothehistograms,theratiooftruepositive(i.e.,probabilityofdetection)isplottedasafunctionoftheratiooffalsepositiveateveryconfidence

Figure 6: Normalized histogram of signal and noise.

Figure 7: Example of ROC curves.

Equation 6

Equation 7

Equation 8

Equation 9

Equation 10

Equation 11

1,2,3and�.Abenchmarkingresultisalsoshowninthetables.Thissectiondiscusseshowthedataareanalyzed.

Analysisofvariance(ANOVA).ANOVAtestsarenecessary iftherearesignificantdifferencesofPDbetweenlevelsforeachfactor.2�This is useful to check if experiments arewell-designed todiscussinfluencesofthefactorsonPDandtoseehowthefactorsinterfereinPD.Somelevelssuchasatargetdepthof30centimetershavebeensettobeverydifficultincomparisonwiththesensorspecificationsbecauseanobjectiveofthetestistomakethelimitationsofthesensorsystemsclear.

Inthefollowingpartofthissection,anexampleisgivenforanANOVAofExperiment2 assuming that an experimental result inTable�isacquiredfromasystemwithnorepetition.Firstthemeanoftheresultsiscalculatedas:

Table 7: PD of eight testees of Experiment 2. Highlighted data of four testees are analyzed as shown in Figure 14.

Table 8: Notion of detection of probability for ANOVA example.

andthemaineffectforeachlevelofthefactorsA,BandCisderivedasfollows:

Next,erroreffectseifori=1,L,�arecalculatedas:

TEST, Continued from page 96

Equation 1

Equation 2

Equation 3

Equation 4

Equation 5

4

Journal of Conventional Weapons Destruction, Vol. 10, Iss. 1 [2006], Art. 44

https://commons.lib.jmu.edu/cisr-journal/vol10/iss1/44

100| research and development |journalofmineaction|2006|august|10.1 10.1|august|2006|journalofmineaction| research and development | 101

Table 10: Eight testees’ result of ANOVA of Experiment 1.

Table 11: Eight testees’ result of ANOVA of Experiment 2.

Figure 11: Factor effects of Experiment 1 with 95% confidence intervals

Figure 13: Averages of PD for Experiment 1. Testees 7, 2,3 and 6 were chosen from each device.

Figure 14: Averages of PD for Experiment 2. Testees 7, 2, 3 and 6 were chosen from each device.

Figure 12: Factor effects of Experiment 2 with 95% confidence intervals.

rating(threshold).AsshowninFigure6,ifasensorfunctionswell,ahistogramoftargets(solidline)isdistributedapartfromthatofnoise(dotted line), and the resulting ROC curve climbs rapidly towardtheupperleft-handcornerofthegraphasshownbythesolidlineinFigure7.Ontheotherhand,ifanothersensorgivesahistogramoftargetsthatisdistributedclosertothatofnoise,theresultingROCcurvegetscloser toadiagonal lineas shownby thedashed line inFigure7.Thismeansthediscriminatingpowerdecreases.OnceROCcurvesareobtained, therearemanymethods to test thedifferencebetweenROCcurves.31

In the experiment, the number of true positives is controlled,butthenumberoffalsepositivesdependsonhowmanyfalsealarmsarereportedbythetestee.Therefore,allthehistogramsdiscussedherearenormalizedbydividingfrequenciesbythetotalnumberofthepopulation.

Experimental ResultsFigure�showsthegroundtruthofthelane2,andFigures9and

10showssubsurfaceimagesfromasensorsystem.Inthiscase,ithasbeen shown that a metal detector can clearly image seven pairs ofType72 surrogates buried flush (Figure 9), and that a GPR sensorcan display seven PMN2 surrogates at a depth of 20 centimeters

Figure 10: Detection image from a GPR sensor.

Figure 9: Detection image from a metal detector.

(�inches)(Figure10),wherethemetaldetectorwasnotabletogetanysignal.Basedonthesekindsofimages,testeeshavederivedtheirdetectionresults,andthissectiondiscussestheexperimentalresults.

Probabilityofdetection.Thenumberof testees is eight, thebreakdown of which is two from MHV with a step-frequencyGPR+MD (MHV #1), two from MHV with a pulse GPR+MD(MHV#2), two from theAdvancedMineSweeperwith a step-frequency GPR+MD, and two from Gryphon with a pulseGPR+MD.TheeightsetsofdatawereanalyzedbyANOVAtoseetheeffectsoffactors.Notethattheorderofthesystemsisnotcon-sistentwithdevices1–�tokeepanonymity.

Tables 10 and 11 show ANOVA results for Experiments 1 and2, respectively, and Figures 11 and 12 show plots of factor effects(i.e.,maineffectsaddedtothemean

µ

with9�-percentconfidence

intervalsderivedinthesamewayasEquation11).InTables10and11,factors,thenullhypothesisofwhichhasbeenrejectedatthelevelofsignificanceof0.0�/0.01,areindicatedby*(0.0�)/**(0.01).Forthosefactors,therehavebeensignificantdifferencesinPDbetweenthelevels,anditcanbesaidthatitismeaningfultodiscusshowthosefactors influence PD and that the test lanes were well-designed toevaluatethesensorsystems.IthasbeenshownthatthereisastrongdependenceofPDontargetdepthand that thedevelopedsystems

still haveproblems for rough andunevenground surface(Figures 11 and 12). Regarding factor A of Experiment2,distance to adjacent target, theANOVAshowed thattherewasnosignificantdifferenceinPDbetweenapairofType72-S surrogates at a 1�-centimeterdistance andtheotherindependentType72-Ssurrogates.

AveragesofPDoffourtestees,thatis,oneeachfromeverysystem,areplottedinFigures13and1�,comparedwiththebenchmarkingresultusingonlyametaldetector.Confidence limits canbe calculated theway thatK.M.Simonsondiscusses in theSandia Report32,33as thenum-berofpopulationforeachlevelisderivedfromTables10and11above.TheseresultsshowedthatthePDfortargetsdeeperthan10centimeterscanbe improvedbycombin-ing GPR with MD. On the other hand, as also shownin Figures 13 and 1�, some of the GPR+MD results inshallow levels were worse than those of metal detectors.Thisisbecausesensorheightabovetheground,whichiscontrolledbymanipulators,ishigherthanthatofmanualscanningofmetaldetectors,andthis isconsideredtobeimprovedbymodifyingthemanipulationalgorithmofaroboticpart.

Lessonslearned.Throughthetestandevaluationpro-cess,many lessonshavebeen learned, someofwhicharelistedbelow:

• Theprovidedcalibrationareashouldhavecontainedlandminesurrogatesforalllevelsoffactors.Coachingatypicalimageforeachlevelwouldmuchimprovethedetectionrate.

• Insomecases(forexample,likeTestee7),highPDshave been accompanied by high false-alarm ratesaround 30 times/square meter,3� and it was alsoproven that confirming the source of false alarmsforGPRismuchmoredifficultthanthoseofmetaldetectors(i.e.,metalfragments).Therefore,anoth-er performance index to penalize these GPR falsealarmswillbeneeded.

• PDindeep levelsof20–30centimeterscanbe im-provedbycombiningGPRwithMDs.

Continued on page 102, TEST

Figure 8: Ground truth of the lane 2; ** shows a pair of Type72 and shows PMN2.

5

Ishikawa et al.: Test and Evaluation of Japanese GPR-based AP Mine Detection Systems Mounted on Robotic Vehicles

Published by JMU Scholarly Commons, 2006

102| research and development |journalofmineaction|2006|august|10.1 10.1|august|2006|journalofmineaction| research and development | 103

T heinternationaldeminingcommunitycontinuestoseekreliable,efficient,andcost-effectivemine-andvegetation-clearanceequip-menttoassistindeminingoperations.TheU.S.Humanitarian

DeminingResearchandDevelopmentProgramisrespondingtothisneed by focusing much of its effort on developing, demonstratingandvalidatingtechnologiesthathelpthedeminingcommunityclearminesandvegetationfaster,saferandmoreefficiently.

Oneof theways inwhichtheHumanitarianDeminingR&DProgrambringseffective,reliable,yetaffordabletechnologiestothefieldisthroughtheadaptationofcommercialoff-the-shelf(COTS)equipment. In particular, one of its most successful strategies isusing a COTS platform and adding tool attachments to create amulti-functioning vehicle. Through past efforts, the HD R&DProgramhasproven the concept thatusing a singleprimemoverwithatoolkitcomprisingawell-thought-outselectionoftoolscanreliablyandrapidlyperformthedeminingtasksoflandpreparation,mineremoval,andareareductionandreclamation,leavinganareaready for quality-assurance proofing. Two such systems currentlyinusebydeminingprogramsaretheSurvivableDeminingTractorandToolsandtheMineClearingSurvivableVehicle(akaMantis).BothsystemsuseCOTSplatformsandavarietyofattachmenttoolstoperformmultipledeminingtasks.

The Survivable Demining Tractor and Tools TheSDTTwasfirstdeveloped in1997and isoneof theearli-

est successes of the HD R&D Program. The system uses a modi-fiedcommercialNewHolland160-90farmtractorfittedwitharmor

Success of Multi-tools in Mine Action: The Survivable Demining Tractor and Tools and the Mine-Clearing Survivable Vehicle

by Tinh Nguyen and Charles Chichester [ U.S. Humanitarian Demining Research and Development Program ]

The authors examine the various equipment and

technologies that allow further effectiveness in

demining achievements. Recent developments

in demining tools allow for greater protection of

deminers, in addition to improved search results.

With technological advancements such as the

Survivable Demining Tractor and Tools and the

Mine-Clearing Survivable Vehicle, the authors

express hope for demining centers worldwide.

plating,optionalsteelwheelsandavarietyofspecializedimplementsusedtoclearheavilyvegetatedareasandsupportvariousdeminingoperationsfromareapreparationtoqualityassurance.Attachmentsincluderollers,magnets,slashers,forestrytoppers,rakes,hedgetrim-mers, sifters, light and heavy cultivators, large and small buckets,largeandsmallgrabs,palletforks,andlightandheavytree-pullers.The system mechanically assists the manual demining process byprovidingdeminersnumeroustoolsandanarmoredplatformfromwhich to perform the most hazardous tasks. The versatility of thesystemallowsdeminerstoworkmoreefficiently.

The SDTT is currently in use by the Thailand Mine ActionCentertoclearvegetationandpreparethelandformanualdemining.From2001through200�,theSDTTclearedover3,�62,310square

ALL

PH

OT

OS

CO

UR

TE

SY

OF

U.S

. HU

MA

NIT

AR

IAN

DE

MIN

ING

RE

SE

AR

CH

A

ND

DE

VE

LOP

ME

NT

PR

OG

RA

M

Mine Survivable Tractor

Grab

TEST, Continued from page 100 EvaluationofFAR.Asdescribedabove,

ROCcurvesareusefultoseethequalifica-tionofsensors,takingintoaccounttradeoffbetweenPDandfalsealarmrate.Table12shows theFARof eight testees for eachofthesixlanesintheexperiment.

Figures 1�a through 1�d show typicalROCcurvesof testees7 and3 for lanes2and�.Lane2has21targetsburiedasshowninFigure�(seepage102),andlane�withroughgroundsurfacehas77targets.Ahori-zontal axisof eachplot shows thenormal-ized FAR, and the number of false alarmscanbederivedbyFARmultipliedbytheto-talnumberofnegativesthatisshownineachplot.InthecaseofFigure1�a,6�percentoftargetsweredetectedwith100-percentcon-fidence,but theother targetsgotmixed in�2�negatives.InFigure1�b,9�percentofthe targetsweredetectedwith100-percentconfidence.Figure1�cforlane�showsthattestee 7 could not discriminate the targetsfrom 73� negatives although the PD was77percent.Ontheotherhand,asshownin

Lane #Device #1 Device #2 Device #3 Device #4

Testee #1 Testee #2 Testee #3 Testee #4 Testee #5 Testee #6 Testee #7 Testee #8

1 11.3 12.4 1.3 1.6 2.7 2.2 20.9 6.0

2 8.5 6.6 1.7 0.7 1.1 1.9 35.0 7.5

3 9.3 8.3 3.2 1.0 2.4 2.1 52.5 6.4

4 15.4 16.7 4.2 1.3 3.9 3.5 36.9 4.6

5 16.0 9.5 0.5 0.7 6.0 2.5 31.9 8.9

6 9.5 12.3 0.9 1.7 4.5 1.7 20.6 8.5

Table 12: False-alarm rate (1/square meter) of eight testees for each lane.

Figure 15b: ROC curve for lane 2 (testee 3). The total number of negatives (fragments, clutters or noise) is shown.

Figure 15a: ROC curve for lane 2 (testee 7). The total number of negatives (fragments, clutters or noise) is shown.

Figure 15c: ROC curve for lane 4 (testee 7). The total number of negatives (fragments, clutters or noise) is shown.

Figure 15d: ROC curve for lane 4 (testee 3). The total number of negatives (fragments, clutters or noise) is shown.

Jun Ishikawa received bachelor’s and master’s degrees in control engineering from the Tokyo Institute of Technology in 1989 and 1991 re-spectively. He joined NEC Corporation in 1991 as a Research Engineer of mechatronics and computer peripher-als. He has been a Research Manager of the R&D Office for Supporting Anti-personnel Mine Detection and Removal Activities for the Japan Science and Technology Agency since 2002.

Mitsuru Kiyota received a bachelor’s of engineering from KEIO University in 1971. He joined Mitsubishi Electric Corporation in 1971. He was engaged in the development of defense equipments. He was transferred to the Japan Science and Technology Agency in 2002 and is now a Research Manager of the R&D Office for Supporting Anti-personnel Mine Detection and Removal Activities.

Katsuhisa Furuta received his Bachelor and Master of Science and doctorate degree in engineering from the Tokyo Institute of Technology in 1962, 1964 and 1967, respectively. He is currently a professor, a member of the Board of Trustees and Director of the 21st Century Center of Excellence Project at Tokyo Denki University and the supervisor of the MEXT/JST Project of Humanitarian Anti-personnel Mine Detection and Removal Activities.

Jun IshikawaResearch ManagerR&D Office for Supporting Anti-personnel Mine Detection and Removal ActivitiesJapan Science and Technology AgencyShibuya-Property-Tokyu Bldg.10F, 1-32-12, HigashiShibuya-ku, Tokyo 150-0011 / JapanTel: +81 3 5778 2001Fax: +81 3 5778 5020E-mail: ishikawa@mine.jst.go.jp Web site: http://www.jstgo/jp/ kisoken/jirai/EN/index-e.html

Mitsuru KiyotaResearch ManagerR&D Office for Supporting

Antipersonnel Mine Detection and Removal Activities

Japan Science and Technology Agency

Katsuhisa FurutaProfessorTokyo Denki University

Figure1�d,testee3,thePDofwhichwas�0percent,detected�0percentoftargetswith100-percentconfidence.Thesekindsofdatahavebeenusedtooptimizetheoperator’sde-cisionthresholdandsensorsensitivities,andtoimprovethesensorperformance.

ConclusionsThrough the test and evaluation, many

lessonshavebeenlearned,andtheseresultswerefedbacktothetesteesforfurtherim-provement. The next step of the project isfield trials in somemine-affected countriestoconfirmtheimprovementsandtoevalu-aterobustnessandcost-effectiveness.

The authors would like to thank all the proj-ect members, especially the principal partners for the trial: Tohoku University, Chiba University, Tokyo Institute of Technology, University of Electro-Communications, Nagoya University, Kyoto University, Osaka University, TADANO Ltd., Mitsui Engineering & Shipbuilding Co., Fuji Heavy Industries Ltd., TAU GIKEN Co. Ltd. and Tokyo Gas Co.

See Endnotes, page 112

6

Journal of Conventional Weapons Destruction, Vol. 10, Iss. 1 [2006], Art. 44

https://commons.lib.jmu.edu/cisr-journal/vol10/iss1/44