ethovision: a versatile video tracking system for automation of

Copyright 2001 Psychonomic Society Inc 398

Behavior Research Methods Instruments amp Computers2001 33 (3) 398-414

Manual Versus AutomatedBehavioral Observation

The behavior of animals is commonly recorded in ei-ther a manual or a semiautomated way Traditionally aresearcher observes the animal if the researcher consid-ers that a certain behavior pattern is displayed he or shenotes the behavior either by writing it down or by enteringthe data into an event-recording program (Noldus 1991Noldus Trienes Hendriksen Jansen amp Jansen 2000)Manual recording of behavior can be implemented witha relatively low investment and for some behaviors itmay be the only way to detect and record their occur-rence however automated observation can provide sig-nificant advantages Behaviors are recorded more reli-ably because the computer algorithm always works in thesame way and the system does not suffer from observerfatigue or drift For instance in contrast to manual ob-servation video tracking carries out pattern analysis ona video image of the observed animals to extract quanti-

tative measurements of the animalsrsquo behavior (for moredetails about how this works see below) Automated ob-servation using video tracking is particularly suitable formeasuring locomotor behavior expressed as spatial mea-surements (distance speed turning etc) that the humanobserver is unable to accurately estimate (Bure Iumlsovaacute Bol-huis amp Bure Iumls 1986 Spruijt Buma van Lochem amp Rous-seau 1998 Spruijt Pitsikas Algeri amp Gispen 1990)Automated observation systems also allow the study ofbehaviors that occur briefly and are then interspersedwith long periods of inaction (Martin Prescott amp Zhu1992) and behaviors that occur over many hours such asdiurnal variation in behavior (Olivo amp Thompson 1988Spruijt amp Gispen 1983)

Historical Development ofAutomated Observation

Technology for automated recording of animal behav-ior and movement has evolved dramatically in the pastdecade Early systems using hard-wired electronics wereable to track only a single animal in highly artificial en-vironments (ie test arenas devoid of any substrate bed-ding or objects besides the animal) For example an openfield can be sampled with a grid of infrared beams eitheras the sole detectors (Clarke Smith amp Justesen 1985Kirkpatrick Schneider amp Pavloski 1991 Robles 1990)or in combination with other methods such as placing aseries of strain gauge transducers under the arena to es-timate the animalrsquos position (Gapenne Simon amp Lannou1990) Various ways have been used to measure the mag-nitude of an animalrsquos motion with types of touch-sensitivesensors A crude estimate of movement can be gained byplacing the animal on a bass loudspeaker and monitoringthe loudspeakerrsquos electrical output when its cone is moved

Many colleagues at Noldus Information Technology have contrib-uted toward the development of EthoVision over the years their hardwork and dedication are acknowledged here Special thanks to BerrySpruijt (Utrecht University) who has been an ongoing source of inspi-ration We are also grateful to those users who provided us with theirresearch design and data for use in the review section of this paper We also thank Niels Cadee Martine Janzen Coen van Kaam Tiffany Mayton-TheCrow and Rudie Trienes for reviewing an earlier versionof the manuscript and providing useful comments Correspondenceshould be addressed to L P J J Noldus Noldus Information Technol-ogy bv PO Box 268 6700 AG Wageningen The Netherlands (e-maillnoldusnoldusnl URL httpwwwnolduscom)

Note The authors have a commercial interest in the product describedin this paper

EthoVision A versatile video tracking systemfor automation of behavioral experiments

LUCAS P J J NOLDUS ANDREW J SPINK and RUUD A J TEGELENBOSCHNoldus Information Technology Wageningen The Netherlands

The need for automating behavioral observations and the evolution of systems developed for thatpurpose is outlined Video tracking systems enable researchers to study behavior in a reliable and con-sistent way and over longer time periods than if they were using manual recording To overcome limi-tations of currently available systems we have designed EthoVision an integrated system for auto-matic recording of activity movement and interactions of animals The EthoVision software ispresented highlighting some key features that separate EthoVision from other systems easy file man-agement independent variable definition flexible arena and zone design several methods of data ac-quisition allowing identification and tracking of multiple animals in multiple arenas and tools for vi-sualization of the tracks and calculation of a range of analysis parameters A review of studies usingEthoVision is presented demonstrating the systemrsquos use in a wide variety of applications Possible fu-ture directions for development are discussed

ETHOVISION VIDEO TRACKING SYSTEM 399

by the rat (Silverman Chang amp Russell 1988) Sensorscan also measure the position of the animal (and hencelocomotion) for instance by changes in the capacitanceof a plate when the animal is in proximity to it (ClarkeSmith amp Justesen 1992) or changes in body resistance(Tarpy amp Murcek 1984) Other comparable detectionmethods have included use of ultrasound (Akaka ampHouck 1980) and microwave Doppler radar (Martin ampUnwin 1980) A modern radar-based ldquoactometerrdquo isable to detect very small movements (which is particu-larly important in studies on insect behavior) and has theadvantage of working in complete darkness (Knoppienvan der Pers amp van Delden 2000) The position of a ratin an open field has been recorded by attaching the rat toa computer joystick via a series of rods attached by a col-lar to the ratrsquos neck (Brodkin amp Nash 1995) Animal be-havior can also be measured using a computerized ver-sion of a Skinner box (Skinner 1938) in which thesubject has to tap a touch-sensitive monitor (Morrison ampBrown 1990 Sahgal amp Steckler 1994) The motion ofindividual limbs can be monitored automatically usingactigraph sensors which detect movement by means ofa piezoelectric accelerometer (May et al 1996) An-other technique for automatic classification of behav-ioral patterns uses sensors to detect the mechanical vi-bration of a platform on which a cage with a mouse isplaced (Schlingmann van de Weerd Baumans Remieamp van Zutphen 1998)

Video TrackingVideo tracking systems were introduced in the early

1990s offering clear advantages of flexibility spatialprecision and accuracy over the various hardware deviceslisted above However with early systems the actual pathof the animal still had to be entered manually by the ex-perimenter by following the track of the animal with acomputer mouse (Pereira amp Oliveira 1994) a joystick(Morrel-Samuels amp Krauss 1990) or a digitizing tabletor similar device (Santucci 1995) Another early methodstill used in some commercially available systems is tofeed the analog video signal to a dedicated video track-ing unit which detects peaks in the voltage of the videosignal (indicating a region of high contrast between thetracked animal and the background) and use this to pro-duce the xy coordinates of the tracked animals This out-put is then fed to the serial port of a computer (KlapdorDulfer amp van der Staay 1996 Vorhees Acuff-SmithMinck amp Butcher 1992) These analog systems have thedisadvantage of being relatively inflexible (dedicated toparticular experimental setups) and can normally trackonly one animal in rather restricted lighting and back-ground conditions

Greater flexibility is achieved by the use of a videodigitizer The first video digitizers were of the columnscan type These had no internal memory of their ownand were able to sample the video signal only at ratherlow rates (Olivo amp Thompson 1988) In contrast a mod-ern frame grabber uses a high-speed analog-to-digital

converter to enable real-time conversion of the entirevideo image to a high-resolution grid of pixels It is alsopossible to acquire digitized video images by first con-verting the video input to a digital video format such asAVI and then using the AVI file as the input for objectdetection (Derry amp Elliot 1997) However this methodhas two disadvantages (1) The AVI file quickly getsvery large (and so only trials of a limited duration can becarried out) and (2) the method does not allow for real-time live data acquisition Alternatively a digital overlayboard can also be used to obtain positional data oftracked animals without the need for a frame grabber(Hartmann Assad Rasnow amp Bower 2000 RasnowAssad Hartmann amp Bower 1997)

A number of modern video tracking systems use framegrabbers to digitize analog video signals This enableshigh-speed data acquisition and therefore tracking ofanimals that are moving relatively fast However mostof these systems have severe limitations They can trackonly one animal in one arena If they can track multipleobjects they cannot identify them individually They tendto require simple backgrounds (in terms of their grayscale values) and can deal with only a limited range ofexperimental setups These systems also cannot handlecolor video

THE ETHOVISION SYSTEM

Development HistoryThe EthoVision video tracking system which we pre-

sent here was developed to overcome the limitations ofthe techniques and systems mentioned above EthoVisionhas been designed as a general-purpose video trackingmovement analysis and behavior recognition systemOn the basis of a high-resolution color video frame grab-ber and flexible software it is a versatile image process-ing system designed to automate behavioral observationand movement tracking on multiple animals simultane-ously against a variety of complex backgrounds

Our group has been involved with the developmentdeployment and support of EthoVision for almost a dec-ade A predecessor of the current EthoVision video track-ing system was first developed in the 1980s at the RudolfMagnus Institute for Neurosciences (RMI) of UtrechtUniversity (Spruijt Hol amp Rousseau 1992) In 1992Noldus Information Technology and the RMI joinedforces in order to develop this prototype into a maturevideo tracking system Over the years development pro-ceeded in close partnership with two universities (UtrechtUniversity and Wageningen University) and three phar-maceutical companies (Bayer AG Germany H Lund-beck AS Denmark Solvay Pharmaceuticals bv TheNetherlands) The first DOS-based implementation wasreleased in 1993 The system has undergone numerousupdates over the years on the basis of feedback fromusers around the world which has resulted in a compre-hensive package for studies of movement and behaviorThe software has recently been redesigned from the

400 NOLDUS SPINK AND TEGELENBOSCH

ground up for 32-bit Windows platforms with a greaterrange of features and a highly interactive graphical userinterface for display of experimental design experimen-tal arena and tracks of movement The first Windowsversion of EthoVision was released at the end of 1999

This paper constitutes the first comprehensive publi-cation about EthoVision (version 22 the latest versionat the time of writing) in the scientific literature Thespecifications and functionality of the DOS programhave never been published which is why many of thefeatures already present in that version are given in somedetail here In the sections below we will outline thefunctionality of the EthoVision software and present anoverview of studies using EthoVision to demonstratethe systemrsquos use in a wide variety of applications

How EthoVision WorksEthoVision is an integrated system comprising vari-

ous software and hardware components (Figure 1) ACCD video camera records the area in which the subject

animals are (the scene) The analog video signal is digi-tized by a frame grabber and passed on to the computerrsquosmemory The frame grabber can either digitize the cam-erarsquos signal directly or take input from a video cassetterecorder The software then analyzes each frame in orderto distinguish the tracked objects from the backgroundon the basis of either their gray scale (brightness) or theirhue and saturation (color) values Having detected theobjects the software extracts the required featuresmdashinthe case of EthoVision 22 this includes the position ofthe mathematical center of each object (center of grav-ity) and its surface area These values are written to atrack file on disk Calculations are carried out on the fea-tures to produce quantified measurements of the ani-malsrsquo behavior For instance if the position of an animalis known for each video frame and the whole series offrames is analyzed the average speed of locomotion ofan animal during an experiment or the distance betweenseveral individually identified animals can be calculatedIn addition if certain regions are identified as being of

Figure 1 Diagram of an EthoVision setup The analog signal from a CCD camera is fed into a frame grabber inside the computerwhich digitizes the image EthoVision then analyzes the signal to produce quantitative descriptions of the tracked animalrsquos behaviorA video cassette recorder and monitor are optional components


interest (eg the center and edges of an open field) theproportion of time spent by the animals in those regionscan also be calculated

ETHOVISION SOFTWARE SPECIFICATIONS

EthoVision may be thought of as consisting of a seriesof modules that function as an integrated whole Thefunctions of each of the modules are outlined below Thesoftware has been written in Microsoft Visual C++(using Visual Studio 60 with STL and MFC) The 32-bitcode has been optimized for Windows 98 NT and 2000

File ManagementInformation in EthoVision is organized at several lev-

els The highest level is called a workspace (ie a con-tainer for one or more experiments) A workspace can beused to keep together experiments that have a specificrelation (a similar setup performed by the same experi-menter etc) An experiment embodies a series of trialscarried out with a particular experimental setup Thedata from one animal collected during a trial (the xy co-ordinates and body surface) is referred to as a track Inaddition to the data files generated by EthoVision theuser can create profiles which contain all the settingsmade for a particular function The EthoVision WorkspaceExplorer enables the user to manage all these files througha simple interface (similar to the Windows Explorer) sothat for instance settings defining the acquisition method(the tracking profile) can be dragged and dropped fromone experiment to another (see Figure 2) An entire ex-periment or workspace can be backed up to a single fileto facilitate both safe keeping of the data and transfer be-tween different computers Tracks (and their associatedindependent variable values) can be imported into the cur-rent experiment from another experiment enabling analy-sis of data across multiple experiments

Experiment DesignIndependent variables As well as tracking objects

EthoVision also allows the researcher to define a com-plete experimental protocol in terms of the independentvariables of an experiment and their values Up to 99 in-dependent variables (eg treatment room temperaturegenetic strain etc) can be defined EthoVision also au-tomatically records a series of system variables such asthe profile (settings file) used time and date of the trialtrial duration and so on These independent variables canbe used to select sort and group data both when plot-ting tracks and when analyzing the data For instanceone can select to plot all the tracks from mice of a par-ticular genotype or calculate the mean time taken to reachthe target of a water maze by control rats relative to treatedrats When the trial is carried out the user can for eachuser-defined independent variable select whether to ac-cept the values predefined in the experimental protocolor enter new values In addition the possible range or aseries of exact possible values can be defined for eachindependent variable (see Figure 3)

Scheduling trials Prior to data acquisition one canpredefine the values of the independent variables for aseries of individual trials one plans to run in an experi-ment In this way the design of the experiment and thetesting can be done before the actual trials are performedEthoVision can thus be used to schedule the trials (ieassist the experimenter in applying the correct treatmentand select the correct animals) During data acquisitionthe values of the independent variables are displayed on

Figure 2 The EthoVision Workspace Explorer The workspaceldquoBRMICrdquo contains three experiments Experiment ldquoF15 fe-male cohort 1rdquo is active (shown in bold) and it contains a seriesof different profiles illustrating the different uses to which thesestored settings may be put


the computer monitor providing immediate feedback onthe characteristics of the trial (which animal is currentlybeing tested what the treatment is etc) Once the ex-periment design has been entered it can be used in mul-tiple experiments thus reducing preparation time for re-peated experiments

Arena and Zone DefinitionArenas Up to 16 enclosures can be placed under one

camera and each enclosure can be treated as a separateindependent replicate (called an arena see Figure 4)The animals in these arenas can be tracked simultane-ously The arena can be of any shape which is easily de-fined using special drawing tools Multiple arena pro-f iles and zone definitions can be def ined for eachexperiment This is particularly handy when for in-stance a water maze experiment is prepared with multi-ple platform positions When setting up the experimentthe user can assign the proper arena definition to eachtrack file before the experiment is started If one definesan arena in the video image in which object trackingtakes place the system ignores parts that do not belongto the defined arena during data acquisition This re-duces the chance that events outside of the experimentalarena will interfere with the actual measurement Byusing different drawing options (rectangle circle poly-gon curve line and freehand) one can quickly draw anyshape of experimental setup and enter it into the com-puter This means that one can use the system for a watermaze experiment an open field test or any other stan-dard test Because the signal from the video camera isdirectly displayed on the computer screen the arena out-lines can be traced with high accuracy

Zones and points of interest The user can define upto 99 regions of interest (called zones) These can beused in the analysis (eg to compute the time spent indifferent parts of a cage) and in automatic start and stopconditions For instance the trial can be stopped auto-matically when a mouse has visited all the arms of aneight-arm maze Zones can be added together to make a

cumulative zone (eg if a maze has more than one tar-get) Zones can also be defined as hidden zones for usewith nest boxes burrows and so on The system assumesthat when an animal disappears from view and it was lastseen adjacent to a hidden zone it is inside the hiddenzone A hidden zone can also double as a normal zoneIf the animal is visible within the boundaries of the zoneEthoVision assumes that it is on top of for example thenest box All zones can be defined and altered either be-fore or after data acquisition allowing iterative explor-atory data analysis of the effects of changing zone posi-tions and shapes Points of interest can also be defined(eg the center of an open field or a novel object that isplaced in a stable)

Calibration The arena can be calibrated so that pa-rameters such as velocity are calculated in meaningfulunits such as millimeters or inches instead of pixelsThere are two calibration methods In standard calibra-tion the user measures a series of lines (eg a meterruler placed on the floor of a cage) If the camera imageis distorted the standard calibration will not be accurate(the standard deviation of the calibration factors is givento enable the user to determine this) and the user can optfor advanced calibration In advanced calibration a gridof points is mapped onto the arena and a series of coor-dinate points are entered giving a calibration that is ac-curate for the entire image even when it is distorted Thisenables the distance-based parameters to be accuratelycalculated when the arena is in a room with a low ceil-ing when an exceptionally large arena necessitates theuse of a very wide angle or fish-eye lens or when thelens is not directly overhead in relation to the arena

Data AcquisitionRunning a trial After the user has defined the sam-

ple rate (up to 30 per second) and trial duration Etho-Vision is ready for data acquisition During tracking thecomputer shows a combination of the recorded path thedetected object shape the current arena zone definitionand the original video image (see Figure 5) Elapsed time

Figure 3 Defining independent variables with EthoVision The system-independent variables are shaded and the user-defined vari-ables have a white background The independent variable properties are also shown for the selected variable (treatment)


number of samples and various other statistics are alsoprovided in real time This allows immediate action if atracking error occurs An optional remote control can beused to start and stop trials at a distance from the com-puter (eg as you stand next to the experimental setup)

Object detection To ensure optimal object detectionin any experimental setup EthoVision offers three dif-ferent object detection methods Gray scaling is the fast-est This method defines the animal as all connectingpixels that are both brighter than a low gray scale thresh-old and darker than a high threshold and it defines thebackground as all other pixels The thresholds can eitherbe set manually by the user or be calculated automaticallyby the program Gray scaling is a fast detection method(allowing the highest sample rate) but it cannot be usedif the same gray scale values are present in both the ani-malrsquos image and the background

The second method subtraction first (before the startof the trial) makes a reference image with no animalspresent and then (during the trial) subtracts the grayscale value of each pixel of the reference image from theequivalent pixel of the live image Any pixels that belong

to objects larger than those defined as noise and thathave a subtracted value other than zero constitute the dif-ference between the reference and the live images due tothe presence of the animal The user can define whetherthe animal has to be lighter or darker than the back-ground or just different This method tolerates more dif-ferences in light intensity across the scene It thus becomespossible to separate an animal with light skin on a gray-ish background from its (black) shadow The method isalso suitable for animals with a heterogeneous fur pat-tern such as hooded rats spotted mice or cows

The third detection method color tracking uses thecolor of the animal (or a marker painted on it) to identifyand track it EthoVision uses the hue and saturation com-ponents of the huendashsaturation-intensity (HSI) color spacemodel to track objects (Spink Buma amp Tegelenbosch2000 see Figure 6) By using both hue and saturationEthoVision can better distinguish objects that are moresimilar in color to each other than if only using hue (egobjects with the same hue but differing saturation val-ues see Figure 7) which is why the system can track asmany as 16 different colors in each arena at once (under

Figure 4 An EthoVision arena definition for a water maze experiment The circular arena (region from which data are acquiredhere labeled ldquoWatermazerdquo) is divided into four zones (labeled ldquoNorthrdquo ldquoEastrdquo ldquoSouthrdquo and ldquoWestrdquo) The small zone ldquoPlatformrdquo de-fines the location of the platform The area outside the arena is ignored leaving data acquisition undisturbed by any movement thattakes place in that part of the scene (eg an operator walking by)


suitable conditions) In addition the use of these twocomplementary detection descriptors makes the objectidentification more robust so that for instance objectscan be tracked more reliably if the light intensity (bright-ness) is uneven across the arena

With all three methods objects that are either smallerthan the animal (eg droppings) or larger than the ani-mal (eg reflections) can be excluded on the basis oftheir size

Object identification EthoVision distinguishes be-tween two animals in the same arena on the basis of sizeTo distinguish two animals that are the same size theuser can partly color one animal to the same grayness asthe background thus reducing its apparent size (see Fig-ure 8) Color tracking can be used to distinguish morethan two animals (up to 16 per arena) If the animals beingtracked are the same color markers can be used to paintblobs on the animals which the system will be able totrack For more details on color tracking see the ldquoObjectDetectionrdquo section above

Image resolution The user can further fine-tune dataacquisition by modifying the image resolution EthoVi-

sion can track at low medium and high resolution Athigh resolution (5 768 3 576 pixels) EthoVision cantrack an animal in an arena with a variable shape and upto 200 times the animalrsquos size By comparison an in-frared detector system typically operates in an arena of05 3 05 m with 12ndash32 beams (though systems havebeen developed with a grid of 24 3 24 beams in an arena2 m in diameter eg Robles 1990) Other options in-clude the possibility to manually adjust contrast bright-ness hue and saturation in order to optimize digitiza-tion of the video signal This is useful when analyzingtrials that were prerecorded on videotape Furthermoreone can scan the complete arena or use a moving scanwindow With the latter option only a limited area aroundthe objectrsquos last position is searched As a result track-ing is not disturbed by occasional moving objects else-where in the arena (eg reflections) When an animal istemporarily out of view EthoVision can be set to resumetracking by automatically repositioning the scan window

Image filtering EthoVision calculates the mathemat-ical point at the center of the digitized picture of the an-imalrsquos body including the tail (eg in a rat or mouse)

Figure 5 Data acquisition in progress during a test of a parasitic wasp searching for hosts in a petri dish In addition to the live videoimage with overlaid zone borders and movement track EthoVision shows tracking statistics (elapsed time number of samples pro-cessor load etc) measurement values of the object (xy coordinates surface area) and characteristics of the trial (independent vari-ables) Image courtesy of J P Monge (University of Tours Tours France)


This means that this point is farther toward the animalrsquosposterior than is correct Furthermore the system willstill detect movement of the animal when only the tail ismoving In order to prevent this the tail can be removedfrom the image using image filtering (erosion) The barsof a cage can also be filtered out using a similar tech-nique (dilation) These filters also help to avoid acciden-tal mix-up of the animal with background noise and to im-prove the computation of body-shape-related parameters

Recording specific behaviors In addition to auto-matically tracking the movement of the animal EthoVi-sion can automatically detect certain behaviors such asrearing and moving (see the Appendix for details) Toallow detection of rearing behavior of rodents using asingle overhead camera EthoVision stores the surfacearea of the animalrsquos body for each sample taken Etho-Vision also allows the researcher to manually record (bykeystroke or mouse click) behaviors that cannot be de-tected automatically (eg sniffing)

Experiment control EthoVision also has a facilityfor automatic experiment control (see Figure 9) Thiscan be used to start or stop data acquisition depending onthe location of the animal For instance the system canbe set to start a trial when a rat enters a water maze andstop it when the rat reaches the platform or to stop thetrial after the animal has been in a user-defined zone formore than a user-defined length of time One can also letthe system perform series of trials with user-defined num-ber of trials and intertrial interval (eg 72 consecutivetrials each with 5-min duration and 55-min intervals in be-tween) which is useful for studies of behavioral rhythms

Selection and Visualization of Track DataThe user can specify data sets at several hierarchical

levels for track analysis and visualization The data canbe selected sorted and grouped by independent vari-ables In addition the track files can be split up (nested)by time Thus a selection of the tracks in an experimentcan be plotted on a single screenmdashfor instance in a ma-trix of treatment level 3 genetic strain Other variablessuch as gender or day of the week can be shown in dif-ferent colors and line styles Furthermore time windowscan be applied (eg just the first 60 sec of the track)The track plots can be displayed with or without the arenaand zones or the background image of the experimentalsetup The plots can be exported as graphic files in a va-riety of standard formats ready for inclusion in presen-tations reports or research articles Figure 10 shows atypical EthoVision visualization of tracks

Figure 6 The HSI color space model The horizontal distancefrom the center of a horizontal plane to its outside represents acolorrsquos saturation (fullness) the angle on that plane represents itshue (color) and the vertical distance (z-axis) represents its inten-sity (brightness)

Figure 7 Object identification based on color Each shape onthe disk (which is displayed in color on screen) represents a vol-ume in HSI color space and the user can drag the shape so thatit includes the range of hues and saturations present in the ani-mal or marker being tracked When the color defined here matchesthe color of the animal or marker the software displays those pix-els in the color selected in the ldquoShow colorrdquo column thus givingvisual feedback as to when the object is correctly identified


EthoVision can replay the recorded tracks on thescreen allowing a detailed and interactive visual analy-sis of the data One can choose for playback to be eitherat the original speed or at a user-defined speed and setthe number of data points that are displayed simultane-ously One can also move through the tracks showingeach time only a selection of data points

Data AnalysisData selection for analysis is carried out in the same

way as for data visualization and in fact the same dataprofile is used so that the data selection made for visu-alization is automatically available for analysis (and viceversa) The user can group tracks by the values of inde-pendent variables (eg the mean velocity can be calcu-lated for all animals that received a particular treatment)and further fine-tune the data using filter settings (egminimal distance moved) to eliminate slight ldquoapparentrdquomovements and using down-sampling steps to eliminateredundant data if a lower sample rate describes the pathbetter A wide range of quantitative measures of behav-ior is available in EthoVision including parameters forlocation and time path shape individual behavioral statesand social interactions and these can be described witha full range of descriptive statistics The parameters sta-tistics and raw data (ie series of xy coordinates andother extracted image features for each individual as afunction of time) can be exported in a variety of formatsfor further analysis and production of graphs in third-

party programs such as Microsoft Excel SAS SPSSThe Observer (Noldus et al 2000) WinTrack (Lipp ampWolfer 2000 Wolfer amp Lipp 1992) or SEE (Drai Ben-jamini amp Golani 2000) More detailed informationabout these parameters is given in the Appendix

REVIEW OF STUDIES USING ETHOVISION

In contrast to some other video analysis systems (egMukhina Bachurin Lermontova amp Zefirov 2001 PanLee amp Lim 1996 Spooner Thomson Hall Morris ampSalter 1994) EthoVision is a generic tool that can beused in a wide variety of setups and applications Thereare of course practical limitations however in princi-ple EthoVision can be used to track animals in any sit-uation where the animals are within sight of a cameraand in a delimited and constant area that is no more than200 times the size of the animal The majority of studiesusing EthoVision are carried out using small rodents butthe system is also used in research on insects fish birdsprimates farm animals and other mammals The reviewbelow gives an overview of the main types of researchusing EthoVision however it is by no means an exhaus-tive list of all current uses of EthoVision

Neuroscience and PharmacologyThe most common application of EthoVision is as a

tool for measuring the behavior of rats and mice thathave had their brain chemistry altered by application of

Figure 8 To distinguish two rats that are the same size one has been partly colored black After ap-plying a gray level threshold only the bright parts of the body remain thus creating an apparent sizedifference


drugs or neurochemicals (eg Ploeger Spruijt amp Cools1994 Sams-Dodd 1995) surgery (eg Eijkenboom ampvan der Staay 1998 van Rijzingen Gispen amp Spruijt1995) or the use of a particular genotype (eg Mini-chiello et al 1999 Spink Tegelenbosch Buma amp Nol-dus 2001) Research of this type often combines theabove techniques (eg testing whether injecting a min-eralocorticoid receptor antagonist can counter the effectsof a lesion Oitzl Josephy amp Spruijt 1993) The brainfunctioning and behavior of rats and mice are often usedas models of human brain functioning and behavior inthe study for instance of the effects of addiction (ChowTomkins amp Sellers 1996 Miczek Nikulina KreamCarter amp Espejo 1999) or sleep deprivation (MeerloOverkamp Benning Koolhaas amp van den Hoofdakker1996) The majority of neuropharmacological studiesuse either the Morris water maze to test spatial learning(Morris 1984 see eg Dalm Grootendorst de Kloetamp Oitzl 2000 Lipp amp Wolfer 2000 see Figures 4 and10) or the open field test to study anxiety andor socialinteraction (eg Spruijt et al 1992 van den Berg Spruijtamp van Ree 1996) These tests are ideally suited to auto-matic data acquisition and analysis with EthoVision theytake place in controlled conditions in a simple definedarea and the resulting behavior can be readily quantified

With the Morris water maze the researcher tests spa-tial memory by measuring how long a rat or mouse takesto swim to a hidden platform When it reaches the plat-form it can stop swimming The animal is first trainedand it normally relies on visual clues to find the platformagain The test thus usually assesses both visual pro-

cessing and spatial memory EthoVision can be used todefine a zone where the platform is and measure the timetaken for the rat or mouse to reach the zone It can alsobe used to assess the reason an animal might ldquofailrdquo thetest For instance some strains of mice hug the walls whenstressed (thigmotaxis) and therefore never find the plat-form This can be an appropriate behavioral response inthe animalsrsquo natural environment and it is important notto confuse this with failing to learn where the platformis because of an error in its learning mechanism (Gasset al 1998 Lipp amp Wolfer 2000) Figure 10 shows thetracks from a Morris water maze experiment

With an open field test the researcher simply placesone or more animals in an open space (usually a squareenclosure with the side about 5ndash10 times as long as thelength of the animalrsquos body) As an animal becomes lessanxious it spends less time by the walls and more timein the open (eg Meerlo et al 1996) Novel objects canbe introduced into the field and the animal will investi-gate the object spending a variable amount of time bythe object depending on the animalrsquos state (WhishawHaun amp Kolb 1999) One of the strengths of EthoVi-sion is that the arena can be divided up into a large num-ber of zones of any size and shape so that the positionsof objects in the field and areas around the object can beaccurately defined and the positions and movements ofthe animal can be measured in relation to those zones Inaddition the zones can be defined after data acquisitionhas occurred so that if areas of interest become apparentonly after an initial analysis these can be investigatedfurther Moving novel objects can also be placed in the

Figure 9 The Experiment Control dialog with three conditions and two actions The prop-erties of the third condition are shown in detail


open field and these can be tracked in the same way thatthe animals are tracked (Spruijt et al 1992) The waythat the animals move in relation to the other objects canthen be quantified by EthoVision and this can be inter-preted in terms of approach avoidance and so on

ToxicologyA standard assessment for the effects (or lack of ef-

fects) of toxins and suspected toxins is the measurementof any changes in behavior under standard conditions(Moser et al 1997) EthoVision is used by a number ofgroups for these measurements For example Hany et al(1999) looked at the behavioral effects on rat pups of ex-posing their mothers to PCBs during the gestation pe-riod The behavior of the rats in an open field was mea-sured using EthoVision The open field was divided intoinner and outer zones and it was shown that rats exposedto PCBs avoided the outer zone Because there was nodifference in the total distance moved between treat-ments this was interpreted as a change in the emotionalstate of the rats rather than an impairment of motorfunction Weinand-Haumlrer Lilienthal Winterhoff andWinneke (1996) also used EthoVision to analyze the ef-fects of PCBs on rat pupsrsquo performance in a Morris watermaze Another typical toxicological application is re-search on the behavioral effects of nicotine (Kim et al

2000) Healing et al (1997) concluded that EthoVisioncan be used to reliably assess changes in motor activityto a standard acceptable to regulatory authorities

EntomologyAlthough EthoVision is most commonly used to track

rats and mice it can also be used successfully to track in-sects and arachnids For example Bouman SimekZemek Dusbabek and Zahradnickova (1999) looked atthe effect of female status on behavioral interactions be-tween male and female Ixodes ricinus ticks whose walk-ing pattern was tracked and analyzed by EthoVision Thelatency to contact between male and female ticks andspeed and duration of movement of males toward fe-males were interpreted as indicators of female attrac-tiveness over short distances and the mean distance be-tween the sexes was interpreted as an index of the overallattractiveness of the female Bouman et al found that themales did not walk randomly but were attracted to the females and that the males were more attracted to the fe-males that were engorged on guinea-pig blood

Kroumlber and Guerin (1999) also studied ticks withEthoVision They analyzed the path moved by Boophilusmicroplus and Ixodes ricinus in relation to a patch ofwater In general as soon as one of the tickrsquos legs touchedthe water the tick rotated and walked away from the

Figure 10 Visualization of an EthoVision water maze experiment (data from Minichiello et al 1999) The Workspace Explorer canbe seen on the left In the main window eight tracks are plotted sorted by trial number (1ndash4) and genetic strain The ldquoPlay Consolerdquo(with VCR-type controls) and ldquoPlay Optionsrdquo dialog box are also shown


water However if the ticks were dehydrated they didnot avoid the water It was hypothesized that ticks haveto strike a balance between their need for water vapor tomaintain water balance and the danger of freezing whencoming into contact with ice crystals in the winter

Parasitic wasps such as Encarsia formosa are of greatcommercial value in biological control of greenhousepests A video tracking system such as EthoVision canbe a useful tool in selecting species suitable under givenconditions for controlling certain pests For exampleDrost Qiu Posthuma-Doodeman and van Lenteren(2000) measured the velocity and turning rate of parisi-toids of Bemisia argentifolia whitefly in order to char-acterize the searching pattern and to identify the bestnatural enemy The wasps were tracked in a petri dish litfrom below which gave a good high-contrast image Theedge of the leaf disk was defined as a zone and was ex-cluded from analysis by nesting (because the boundarycauses unnatural 180ordm turns) The data will be used tocreate a simulation model of host-searching behavior foreach of the species studied Krips Kleijn Wilems Golsand Dicke (1999) carried out a similar study to deter-mine the effects of leaf hair density on the searching ef-ficiency of Phytoseiulus persimilis (a predatory mite usedfor biological control of spider mites) although a speciesmay have been successfully used to control the pest on aplant with smooth leaves it may be quite unable to copewith leaves with dense hairs Movement parameters havethe potential of becoming a standard measurement in thequality control of mass-reared insects (van Schelt BumaMoskal Smit amp van Lenteren 1995) EthoVision has alsobeen used in studies of behavioral aging in Drosophila(Le Bourg amp Minois 1999)

Animal WelfareEthoVision has been used to study stress effects on a

variety of species including captive animals such as farmanimals (Bokkers amp Koene 2000 IumlSustr IumlSpinka amp New-berry 2000) laboratory animals (Ruis et al 1999) andzoo animals (Fuchs Kramer Hermes Netter amp Hiemke1996 van Kampen Schmitt Hiemke amp Fuchs 2000)For example battery hens were tracked with EthoVisionas they walked toward food The latency to pecking thefood and the speed of walking toward the food were mea-sured and used to assess the animalsrsquo motivation andability to walk (Bokkers amp Koene 2000)

IumlSustr et al (2000) marked the front and rear of pigswith different colors and tracked these with EthoVisionThey were then able to use the EthoVision data to ana-lyze pig play and pig fights in terms of mutual spatialposition and orientation First the data were correctedfor samples missed when for example the head wasdipped down and out of sight of the camera Then theactual positions of the heads and snouts were calculatedusing the positions of the markers on the pigs and theknown distances from the markers to the heads andsnouts These positions were then used to calculate howclose the pigs were to each other and which pig was ldquoac-tiverdquo (ie its snout was nearer to the body of the other

one) Finally the angle of the two bodies and point ofcontact were calculated

Fish BehaviorOne of the first uses of an early predecessor of the cur-

rent EthoVision program was to study locomotor activityin Arctic char (Winberg Nilsson Spruijt amp Houmlglund1993) This species is still being studied with EthoVisionThe char are placed in a fluvarium the upstream end ofwhich is divided into two halves Each half receives waterfrom a separate source Every 90 min the tanks areswitched and the fish are tracked for the latter half ofeach 90-min period The tanks are backlit with light witha wavelength of less than 750 nm which is detectable bya monochrome camera but not by the fish The analysisis based on the frequency that the fish are in each half ofthe fluvarium (which is defined as an EthoVision zone)which is in effect a Y-maze Bjerselius Olseacuten and Zheng(1995a 1995b) found that male fish avoided the half ofthe fluvarium that had elevated levels of the female hor-mone indicating that the female was not spawning If in-stead of a semiochemical being added the two halves ofa fluvarium were attached to tanks containing other charthe char in the fluvarium had a different frequency of be-ing in the two halves of the fluvarium depending onwhether the water came from fish that were siblings (Ol-seacuten amp Winberg 1996) and whether the siblings had anidentical major histocompatability complex (OlseacutenGrahn Lohm amp Langefors 1998)

Ylieff Sanchez-Colero Poncin Voss and Ruwet (2000)were able to use EthoVision to track damsel fish (Chromischromis) marked with colored pearls to quantify vari-ables that cannot be measured accurately by direct ob-servation methods Having determined that the coloredmarker did not affect velocity distance moved or time invarious zones they investigated the effects of water tem-perature and fish density on the behavior of the fish Theyfound that at high temperatures the f ish swam fasterwhen they were near the surface which was hypothesizedto be an adaptation to escape from predatory birds

Van Ginneken et al (1997) used EthoVision to com-bine calorimetry of tilapia with analysis of activity Inthis way the effects of various movements on the in-creased metabolic rate of the fish under various light andoxygen conditions could be analyzed

DISCUSSION

As can be seen from the ldquoReviewrdquo section of this paperone of the strengths of EthoVision is that it is flexibleenough to be used in a wide variety of experimental set-ups and applications and with a wide-ranging assortmentof species Some other systems have been specificallydesigned for a particular experiment such as the Morriswater maze (eg Mukhina et al 2001 Spooner et al1994) whereas EthoVision can work with any shape ofarena with a large variety of backgrounds and lightingconditions The ability of EthoVision to track multipleanimals in an arena means not only that social interac-


tions can be studied (Rousseau Spruijt amp Gispen 1996Sams-Dodd 1995 Sgoifo Kole Buwalka de Boer ampKoolhaas 1998 Spruijt et al 1992) but that separateparts of the animals can be marked and tracked indepen-dently ( IumlSustr et al 2000) The sophisticated range of dataselection options and the wide array of variables thatEthoVision can calculate to quantify the animalsrsquo be-havior (parameters) mean that the user is able to use thesystem to produce accurate descriptors of complex be-haviors in diverse situations and is able to gain an over-view of the behavior from the visualization of the ani-malsrsquo tracks

Future DevelopmentsOf course despite the power and flexibility of a sys-

tem such as EthoVision there is always room for furtherdevelopment and improvement We are currently in-volved in and in the past have carried out a number ofspecial projects developing customized versions of Etho-Vision Which of these customizations are integratedinto the standard software depends on both the technicalfeasibility and interest in the application The followingaspects are some indications of the way in which theEthoVision system might be developed in the future

Support of digital video At the moment the Etho-Vision software takes its input from an analog videocamera the signal of which is digitized by a frame grabberA logical development in the future is that EthoVisionwould be able to take direct input from either digital cam-eras or digital video files

Analysis of thermographic images The EthoVisionsystem is designed to measure animalsrsquo behavior and notthe physiological changes that may have influenced thatbehavior Of course it is used in many studies with aphysiological aspect (as in many of the studies cited inthis ldquoReviewrdquo section) and physiological data gatheredat the same time are commonly analyzed together withthe behavioral data A specialized application of Etho-Vision can use the system to directly measure a physio-logical parameter It is well known that stress can causechanges in body temperature of various homeothermicspecies such as rats and mice (Clement Mills amp Brock-way 1989) However the very act of inserting a rectalprobe or implanting a biotelemetry device (both are com-monly used techniques to measure an animalrsquos body tem-perature) stresses the animal and raises its temperature(van der Heyden Zethof amp Olivier 1997) As an alter-native Houx Buma and Spruijt (2000) have developeda noninvasive method to measure body temperature Aninfrared thermographic video camera is connected to acustomized version of EthoVision and the gray scalevalue (ie brightness) of each pixel in the image is pro-portional to the animalrsquos temperature at that locationThus the animalrsquos behavior and temperatures of differ-ent regions of the body can all be measured synchronouslyFurther research is necessary before this technique canbecome a standard tool

Synchronization of behavioral data (video) andphysiological signals In order to carry out a full statis-

tical analysis of variables derived from both EthoVisiontrack files and various other data it is necessary for thevarious data streams to be synchronized At the time ofthis writing we are carrying out a research project to-gether with several partners to resolve this issue (Hooge-boom 2000)

Automatic detection of complex behaviors In itscurrent implementation EthoVision is able to detectwhether an animal is moving or whether a rodent is rear-ing but (in common with other commercial systems) itcannot automatically detect other body postures or be-havioral patterns (unless of course it is possible to de-fine them in terms of the existing parameters) Researchon automatic classification of body postures and behav-ioral acts in digitized video sequences was already in pro-gress more than two decades ago (Kernan et al 1980)but has not to our knowledge found its way into a com-mercial product so far Recently however considerableprogress has been made with the use of model-based pat-tern recognition statistical classification and neuralnetworks to automatically detect rodent behaviors suchas sitting grooming and stretched attend (Rousseauvan Lochem Gispen amp Spruijt 2000 van LochemBuma Rousseau amp Noldus 1998) Further refinement ofthese and other algorithms (eg Heeren amp Cools 2000Twining Taylor amp Courtney 2001) is necessary beforethey can be incorporated into the EthoVision systemThis development is likely to benefit from recent workdone on automated coding of human facial expressionsusing digital image processing active shape modelingand neural network techniques for pattern classification(Ekman amp Friesen 1978 Lanitis Taylor amp Cootes 1997Tian Kanade amp Cohn 2001)

Tracking large and indeterminate numbers of ani-mals Tracking large and continually varying numbersof animals especially if crowded close together calls fortechniques quite different from the object identificationmethods used by EthoVision Some progress has beenmade using techniques such as track segmentation andreconstruction (Buma Moskal amp Liang 1998) and ac-tive modeling and prediction of the animalsrsquo shapes (Bul-pitt Boyle amp Forbes 2000 Sergeant Boyle amp Forbes1988)

AvailabilityEthoVision is commercially available from Noldus In-

formation Technology and various international distrib-utors Readers can contact the first author for more in-formation or visit the EthoVision homepage on the Web(httpwwwnolduscomproductsethovision )

REFERENCES

Akaka W H amp Houck B A (1980) The use of an ultrasonic mon-itor for recording locomotor activity Behavior Research Methods ampInstrumentation 12 514-516

Bjerselius R Olseacuten K H amp Zheng W B (1995a) Behaviouraland endocrinological responses of mature male goldfish to the sexpheromone 17a20b-dihydroxy-4-pregnen-3-one in the water Jour-nal of Experimental Biology 198 747-754


Bjerselius R Olseacuten K H amp Zheng W B (1995b) Endocrine go-nadal and behavioral responses of male crucian carp to the hormonalpheromone 17a20b-dihydroxy-4-pregnen-3-one Chemical Senses20 221-230

Bokkers E amp Koene P (2000) Motivation and ability to walk inbroilers and layer chicks on two diets In Proceedings of the 34th In-ternational Congress of the International Society of Applied Ethology(Florianopolis 17ndash20 October 2000) p 115 Florianopolis UFSC

Bouman E A P Simek P Zemek R Dusbabek F amp Zahrad-nickova H (1999) Methods in the study of sexual behaviour of thetick Ixodes ricinus In Abstracts of the 13th European Meeting of theSociety for Vector Ecology (6-11 September 1999) p84 CoronaCA Society for Vector Ecology

Brodkin J amp Nash J F (1995) A novel apparatus for measuring ratlocomotor behavior Journal of Neuroscience Methods 57 171-176

Bulpitt A J Boyle R D amp Forbes J M (2000) Monitoring be-havior of individuals in crowded scenes In Proceedings of Measur-ing Behavior 2000 3rd International Conference on Methods andTechniques in Behavioral Research (Nijmegen 15ndash18 August 2000)pp 28-30 Wageningen the Netherlands Noldus Information Tech-nology

Buma M O S Moskal J R amp Liang D (1998) EthoVision Multi-Pro Improved animal identification during automatic multi-objecttracking In Proceedings of Measuring Behavior rsquo98 2nd Interna-tional Conference on Methods and Techniques in Behavioral Re-search (Groningen 18ndash21 August 1998) pp 103-104 Wageningenthe Netherlands Noldus Information Technology

Bure Iumlsovaacute O Bolhuis J J amp Bure Iumls J (1986) Differential effectsof cholinergic blockade on performance of rats in the water tank nav-igation task and in a radial water maze Behavioral Neuroscience100 476-482

Chow B Tomkins D M amp Sellers E M (1996) Developing an ani-mal model for drug craving Behavioural Pharmacolog y 7(Suppl 1)16

Clarke R L Smith R F amp Justesen D R (1985) An infrared de-vice for detecting locomotor activity Behavior Research MethodsInstruments amp Computers 17 519-525

Clarke R L Smith R F amp Justesen D R (1992) A programmableproximity-contact sensor to detect location or locomotion of animalsBehavior Research Methods Instruments amp Computers 24 515-518

Clement J C Mills P amp Brockway B P (1989) Use of teleme-try to record body temperature and activity in mice Journal of Phar-macological Methods 21 129-140

Dalm S Grootendorst S de Kloet E R amp Oitzl M S (2000)Quantification of swim patterns in the Morris water maze BehaviorResearch Methods Instruments amp Computers 32 134-139

Derry J F amp Elliott C J H (1997) Automated 3-D tracking of avideo-captured movement using the example of an aquatic molluskBehavior Research Methods Instruments amp Computers 29 353-357

Drai D Benjamini Y amp Golani I (2000) SEE Software for theexploration of exploration In Proceedings of Measuring Behavior2000 3rd International Conference on Methods and Techniques in Be-havioral Research (Nijmegen 15ndash18 August 2000) pp 89-90 Wa-geningen the Netherlands Noldus Information Technology

Drost Y C Qiu Y T Posthuma-Doodeman C J A M amp vanLenteren J C (2000) Comparison of searching patterns of fiveparasitoid species of Bemisia argentifolii Bellows and Perring (HomAleyrodidae) Journal of Applied Entomology 124 105-112

Eijkenboom M amp van der Staay F J (1998) Effects of brain lesionson Morris water maze performance in rats Testing the reliability ofan automatic video tracking system In Proceedings of MeasuringBehavior lsquo98 2nd International Conference on Methods and Tech-niques in Behavioral Research (Groningen 18ndash21 August 1998)pp 137-138 Wageningen the Netherlands Noldus InformationTechnology

Ekman P amp Friesen W V (1978) The facial action coding systemA technique for the measurement of facial movement San FranciscoConsulting Psychologists Press

Fuchs E Kramer M Hermes B Netter P amp Hiemke C (1996)Psychological stress in tree shrews Clomipramine counteracts behav-ioral and endocrine changes Pharmacology Biochemistry amp Behav-ior 54 219-228

Gapenne O Simon P amp Lannou J (1990) A simple method forrecording the path of a rat in an open field Behavior Research Meth-ods Instruments amp Computers 22 443-448

Gass P Wolfer D P Balschun D Rudolph D Frey U LippH P amp Schuumltz G (1998) Deficits in memory tasks of mice withCREB mutations depend on gene dosage Learning amp Memory 5274-288

Hany J Lilienthal H Roth-Haumlrer A Ostendorop G Hein-zow B amp Winneke G (1999) Behavioral effects following singleand combined maternal exposure to PCB 77 (343cent-tetrabiphenyl)and PCB 47 (242cent4cent-tetrabiphenyl) in rats Neurotoxicology amp Ter-atology 21 147-156

Hartmann M J Assad C Rasnow B amp Bower J M (2000) Ap-plications of video mixing and digital overlay to neuroetholog y Meth-ods A Companion to Methods in Enzymology 21 385-391

Healing G Harvey P W McFarlane M Buss N A P S Mal-lyon B A amp Cockburn A (1997) Assessment of motor activityin regulatory neurotoxicity studies Validation of the EthoVision videotracking system in rats treated with amphetamine and chlorproma-zine Toxicology Methods 7 279-287

Heeren D J amp Cools A R (2000) Classifying postures of freelymoving rodents with the help of Fourier descriptors and a neural net-work Behavior Research Methods Instruments amp Computers 3256-62

Hoogeboom P J (2000) Data synchronisation through post-processingIn Proceedings of Measuring Behavior 2000 3rd International Con-ference on Methods and Techniques in Behavioral Research (Nij-megen 15ndash18 August 2000) pp 147-150 Wageningen the Nether-lands Noldus Information Technology

Houx B B Buma M O S amp Spruijt B M (2000) Non-invasivetracking with automated infrared thermography Measuring insideout In Proceedings of Measuring Behavior 2000 3rd InternationalConference on Methods and Techniques in Behavioral Research (Nij-megen 15ndash18 August 2000) pp 151-152 Wageningen the Nether-lands Noldus Information Technology

Kernan W J Jr Higby W J Hopper D L Cunningham WLloyd W E amp Reiter L (1980) Pattern recognition of behavioralevents in the nonhuman primate Behavior Research Methods amp In-strumentation 12 524-534

Kim H-C Jhoo W-K Kim W-K Suh J-H Shin E-J Kato Kamp Ko K-H (2000) An immunocytochemical study of mitochondrialmanganese-superoxide dismutase in the rat hippocampus after kainateadministration Neuroscience Letters 281 65-68

Kirkpatrick T Schneider C W amp Pavloski R (1991) A com-puterized infrared monitor for following movement in aquatic animalsBehavior Research Methods Instruments amp Computers 23 16-22

Klapdor K Dulfer B G amp van der Staay F J (1996) A computer-aided method to analyse foot print patterns of rats miceand humans In Proceedings of Measuring Behavior lsquo96 Interna-tional Workshop on Methods and Techniques in Behavioral Research(Utrecht 16ndash18 October 1996) p 60 Wageningen the Netherlands Noldus Information Technology

Knoppien P van der Pers J N C amp van Delden W (2000) Quan-tification of locomotion and the effect of food deprivation on loco-motor activity in Drosophila Journal of Insect Behavior 13 27-43

Krips O E Kleijn P W Wilems P E L Gols G J Z ampDicke M (1999) Leaf hairs influence searching efficiency and pre-dation rate of the predatory mite Phytoseiulus persimilis (Acari Phy-toseiidae) Experimental amp Applied Aracology 23 119-131

Kroumlber T amp Guerin P M (1999) Ioxid ticks avoid contact with liq-uid water Journal of Experimental Biology 202 1877-1883

Lanitis A Taylor C J amp Cootes T F (1997) Automatic inter-pretation and coding of face images using flexible models IEEE Trans-actions on Pattern Analysis amp Machine Intelligence 19 743-756

Le Bourg E amp Minois N (1999) A mild stress hypergravity expo-sure postpones behavioral aging in Drosophila melanogaster Ex-perimental Gerontology 34 157-172

Lipp H P amp Wolfer D P (2000) Assessing behavior memory andlearning of genetically modified mice by factor analysis based ontrack analysis using EthoVision and WinTrack In Proceedings ofMeasuring Behavior 2000 3rd International Conference on Meth-ods and Techniques in Behavioral Research (Nijmegen 15ndash18 Au-


gust 2000) pp 199-200 Wageningen the Netherlands Noldus In-formation Technology

Martin B R Prescott W R amp Zhu M (1992) Quantification ofrodent catalepsy by a computer-imaging technique PharmacologyBiochemistry amp Behavior 43 381-386

Martin P H amp Unwin D M (1980) A microwave Doppler radar ac-tivity monitor Behavior Research Methods amp Instrumentation 12517-520

May C H Sing H C Cephus R Vogel S Shaya E K amp Wag-ner H N Jr (1996) A new method of monitoring motor activityin baboons Behavior Research Methods Instruments amp Computers28 23-26

Meerlo P Overkamp G J F Benning M A Koolhaas J M ampvan den Hoofdakker R H (1996) Long-term changes in openfield behavior following a single social defeat in rats can be reversedby sleep deprivation Physiology amp Behavior 60 115-119

Miczek K A Nikulina E Kream R M Carter G amp EspejoE F (1999) Behavioral sensitization to cocaine after a brief socialdefeat stress c-fos expression in the PAG Psychopharmacology 141 225-234

Minichiello L Korte M Wolfer D P Kuumlhn R Unsicker KCestari V Rossi-Arnaud C Lipp H P Bonhoeffer T ampKlein R (1999) Essential role for TrkB receptors in hippocampus -mediated learning Neuron 24 401-414

Morrel-Samuels P amp Krauss R M (1990) Cartesian analysis Acomputerndashvideo interface for measuring motion without physical con-tact Behavior Research Methods Instruments amp Computers 22466-470

Morris R (1984) Development of a water-maze procedure for studyingspatial learning in the rat Journal of Neuroscience Methods 11 47-60

Morrison S K amp Brown M F (1990) The touch screen system inthe pigeon laboratory An initial evaluation of its utility Behavior Re-search Methods Instruments amp Computers 22 123-126

Moser V C Becking G C Cuomo V Frantik E Kulig B MMacPhail R C amp Tilson H A (1997) The IPCS collaborativestudy on neurobehavioral screening methods IV Control data Neuro-toxicology 18 947-968

Mukhina T V Bachurin S O Lermontova N N amp Zefirov N S (2001) Versatile computerized system for tracking and analy-sis of water maze tests Behavior Research Methods Instruments ampComputers 33 371-380

Noldus L P J J (1991) The Observer A software system for col-lection and analysis of observational data Behavior Research MethodsInstruments amp Computers 23 415-429

Noldus L P J J Trienes R J H Hendriksen A H M JansenH amp Jansen R G (2000) The Observer Video-Pro New softwarefor the collection management and presentation of time-structureddata from videotapes and digital media files Behavior Research Meth-ods Instruments amp Computers 32 197-206

Oitzl M S Josephy M amp Spruijt B M (1993) An ACTHMSH(4-9)analog counteracts the behavioral effects of a mineralocorticoid recep-tor antagonist Pharmacology Biochemistry amp Behavior 44 447-450

Olivo R F amp Thompson M C (1988) Monitoring animalsrsquo move-ments using digitized video images Behavior Research Methods In-struments amp Computers 20 485-490

Olseacuten K H Grahn M Lohm J amp Langefors Aring (1998) MHCand kin discrimination in juvenile Arctic char Salvelinus alpinus (L)Animal Behaviour 56 319-327

Olseacuten K H amp Winberg S (1996) Learning and sibling odor prefer-ence in juvenile Artic char Salvelinus alpinus (L) Journal of Chem-ical Ecology 22 773-786

Pan W H T Lee C-R amp Lim L-H (1996) A new video path ana-lyzer to monitor travel distance rearing and stereotypic movementof rats Journal of Neuroscience Methods 70 39-43

Pereira P amp Oliveira R F (1994) A simple method using a singlevideo camera to determine the three-dimensional position of a fish Be-havior Research Methods Instruments amp Computers 26 443-446

Ploeger G E Spruijt B M amp Cools A R (1994) Spatial local-ization in the Morris water maze in rats Acquisition is affected byintra-accumbens injections of the dopaminergic antagonist haloperidol Behavioral Neuroscience 108 927-934

Rasnow B Assad C Hartmann M J amp Bower J M (1997) Ap-plications of multimedia computers and video mixing to neuroethol -ogy Journal of Neuroscience Methods 76 83-91

Robles E (1990) A method to analyze the spatial distribution of be-havior Behavior Research Methods Instruments amp Computers 22540-549

Rousseau J B I Spruijt B M amp Gispen W H (1996) Automatedobservation of multiple individually identified rats In Proceedings ofMeasuring Behavior rsquo96 International Workshop on Methods andTechniques in Behavioral Research (Utrecht 16ndash18 October 1996)pp 86-87 Wageningen the Netherlands Noldus Information Tech-nology

Rousseau J B I van Lochem P B A Gispen W H amp SpruijtB M (2000) Classification of rat behavior with an image-processingmethod and a neural network Behavior Research Methods Instru-ments amp Computers 32 63-71

Ruis M A W te Brake J H A Buwalda B de Boer S FMeerlo P Korte S M Blokhuis H J amp Koolhaas J H (1999)Housing familiar male wildtype rats together reduces the long-termadverse behavioural and physiological effects of social defeat Psycho-neuroendocrinology 24 285-300

Sahgal A amp Steckler T (1994) TouchWindows and operant be-haviour in rats Journal of Neuroscience Methods 55 59-64

Sams-Dodd F (1995) Automation of the social interaction test by avideo tracking system Behavioral effects of repeated phenocyclidinetreatment Journal of Neuroscience Methods 59 157-167

Santucci A C (1995) An affordable computer-aided method for con-ducting Morris water maze testing Behavior Research Methods In-struments amp Computers 27 60-64

Schlingmann F van de Weerd H A Baumans V Remie R ampvan Zutphen L F M (1998) A balance device for the analysis ofbehavioural patterns of the mouse Animal Welfare 7 177-188

Sergeant D M Boyle R D amp Forbes J M (1988) Computer vi-sual tracking of poultry Computers amp Electronics in Agriculture 211-18

Sgoifo A Kole M H Buwalka B de Boer S F amp KoolhaasJ M (1998) Video tracking of social behaviors and telemetered ECGmeasurements in colony housed rats In Proceedings of MeasuringBehavior lsquo98 2nd International Conference on Methods and Tech-niques in Behavioral Research (Groningen 18ndash21 August 1998)pp 258-259 Wageningen the Netherlands Noldus Information Tech-nology

Silverman R W Chang A S amp Russell R W (1988) A microcomputer-controlled system for measuring reactivity in smallanimals Behavior Research Methods Instruments amp Computers 20495-498

Skinner B F (1938) The behavior of organisms New York Appleton-Century-Crofts

Spink A J Buma M O S amp Tegelenbosch R A J (2000) Etho-Vision color identification A new method for color tracking usingboth hue and saturation In Proceedings of Measuring Behavior2000 3rd International Conference on Methods and Techniques inBehavioral Research (Nijmegen 15ndash18 August 2000) pp 295-297Wageningen the Netherlands Noldus Information Technology

Spink A J Tegelenbosch R A J Buma M O S amp Noldus L P J J (2001) The EthoVision video tracking systemmdashA tool forbehavioral phenotyping of transgenic mice Physiology amp Behavior73 719-730

Spooner R I W Thomson A Hall J Morris R G M ampSalter S H (1994) The Atlantis Platform A new design and fur-ther developments of Bure Iumlsovaacutersquos on-demand platform for the watermaze Learning amp Memory 1 203-211

Spruijt B M Buma M O S van Lochem P B A amp RousseauJ B I (1998) Automatic behavior recognition What do we want torecognize and how do we measure it In Proceedings of Measuring Be-havior lsquo98 2nd International Conference on Methods and Techniquesin Behavioral Research (Groningen 18ndash21 August 1998) pp 264-266 Wageningen the Netherlands Noldus Information Technology

Spruijt B M amp Gispen W H (1983) Prolonged animal observationsby use of digitized video displays Pharmacology Biochemistry ampBehavior 19 765-769


Spruijt B M Hol T amp Rousseau J B I (1992) Approach avoid-ance and contact behavior of individually recognized animals auto-matically quantified with an imaging technique Physiology amp Behav-ior 51 747-752

Spruijt B M Pitsikas N Algeri S amp Gispen W H (1990)Org2766 improves performance of rats with unilateral lesions in thefimbria fornix in a spatial learning task Brain Research 527 192-197

IumlSustr P IumlSpinka M amp Newberry R C (2000) Automatic computeranalysis of pig play In Proceedings of Measuring Behavior 2000 3rdInternational Conference on Methods and Techniques in BehavioralResearch (Nijmegen 15ndash18 August 2000) pp 307-308 Wagenin-gen the Netherlands Noldus Information Technology

Tarpy R M amp Murcek R J (1984) An electronic device for detectingactivity in caged rodents Behavior Research Methods Instrumentsamp Computers 16 383-387

Tian Y L Kanade T amp Cohn F F (2001) Recognizing action unitsfor facial expression analysis IEEE Transactions on Pattern Analy-sis amp Machine Intelligence 23 97-116

Twining C J Taylor C J amp Courtney P (2001) Robust trackingand posture description for laboratory rodents using active shape mod-els Behavior Research Methods Instruments amp Computers 33

van den Berg C L Spruijt B M amp van Ree J M (1996) En-dogenous opioids and opiate antagonists in the development of socialbehaviour Behavioural Pharmacology 7(Suppl 1) 115

van der Heyden J A M Zethof T J J amp Olivier B (1997)Stress-induced hyperthermia in singly housed mice Physiology ampBehavior 62 463-470

van Ginneken V J T Addink A D F van den ThillartG E E J M Korner F Noldus L amp Buma M (1997) Meta-bolic rate and level of activity determined in tilapia (Oreochromismossambicus Peters) by direct and indirect calorimetry and videomonitoring Thermochimica Acta 291 1-13

van Kampen M Schmitt U Hiemke C amp Fuchs E (2000) Diaze-pam has no beneficial effects on stress-induced behavioral and en-docrine changes in male tree shrews Pharmacology Biochemistry ampBehavior 65 539-546

van Lochem P B A Buma M O S Rousseau J B I amp NoldusL P J J (1998) Automatic recognition of behavioral patterns of ratsusing video imaging and statistical classification In Proceedings ofMeasuring Behavior lsquo98 2nd International Conference on Methods

and Techniques in Behavioral Research (Groningen 18ndash21 August1998) pp 203-204 Wageningen the Netherlands Noldus Informa-tion Technology

van Rijzingen I M S Gispen W H amp Spruijt B M (1995) Ol-factory bulbectomy temporarily impairs Morris maze performanceAn ACTH(4-9) analog accelerates return of function Physiology ampBehavior 58 147-152

van Schelt J Buma M O S Moskal J Smit J amp van Len-teren J C (1995) EntoTrack Using video imaging to automate thequality control of mass-reared arthropods [Abstract] In Proceedingsof IOBC Workshop Quality Control (Santa Barbara 9ndash12 October1995)

Vorhees C V Acuff-Smith K D Minck D R amp Butcher R E(1992) A method for measuring locomotor behavior in rodents Contrast-sensitive computer-controlled video tracking activity as-sessment in rats Neurotoxicology amp Teratology 14 43-49

Weinand-Haumlrer A Lilienthal H Winterhoff H amp Win-neke G (1996) Exposure to a coplanar PCB congener or PTU andbehavioral measurement in rat offspring In Proceedings of Measur-ing Behavior lsquo96 International Workshop on Methods and Tech-niques in Behavioral Research (Utrecht 16ndash18 October 1996)p 108 Wageningen the Netherlands Noldus Information Technol-ogy

Whishaw I Q Haun F amp Kolb B (1999) Analysis of behavior inlaboratory rodents In U Windhorst amp H Johansson (Eds) Moderntechniques in neuroscience (pp 1243-1268) Berlin Springer-Verlag

Winberg S Nilsson G Spruijt B M amp Houmlglund U (1993)Spontaneous locomotor activity in Arctic char measured by a com-puterized imaging technique Role of brain serotonergic activityJournal of Experimental Biolology 179 213-232

Wolfer D P amp Lipp H P (1992) A new computer program for de-tailed off-line analysis of swimming navigation in the Morris watermaze Journal of Neuroscience Methods 41 65-74

Ylieff M Y Sanchez-Colero C Poncin P Voss J amp RuwetJ C (2000) Measuring effects of different temperatures on swim-ming activity and social behavior in groups of Mediterranean marinefish with the EthoVision Color-Pro video tracking system In Pro-ceedings of Measuring Behavior 2000 3rd International Conferenceon Methods and Techniques in Behavioral Research (Nijmegen15ndash18 August 2000) pp 350-351 Wageningen the NetherlandsNoldus Information Technology

APPENDIXEthoVisionrsquos Analysis Parameters

Each selected parameter (eg velocity) is calculated for eachindividual sample The user selects which statistic is required(eg mean) and the combination of the two is displayed (egmean velocity) per track or per group (eg the mean of the totaldistance moved for all animals that received a given treatment)Input filters can be applied to the data to down-sample the pointsandor exclude samples where there is just an apparent movementdue to for example breathing Most of the parameters (eg dis-tance moved and velocity) are continuous variables but some arestate variables because they are either true or false (in zone rela-tive movement moving rearing and manually recorded behaviors)

Location and TimeDistance moved The length of the vector connecting two

sample points (ie the distance of the center of gravity of thetracked animal between one sample and the next) This is cal-culated using Phythagorasrsquo theorem (a2 + b2 5 c2)

Velocity Distance moved per time unit (ie speed)Distance to point The distance between the center of grav-

ity of the tracked animal and a location inside or outside the arenadefined by the user

Distance to zone center The shortest distance between thecenter of gravity of the tracked animal and the center of a user-defined zone

Distance to zone border The shortest distance between thecenter of gravity of the tracked animal and the border of a user-defined zone

In zone Whether or not the animal is in a particular zoneZones can be redrawn at any time They can also be added to-gether to form cumulative zones and can be hidden zones foruse with burrows or nest boxes

These basic parameters are used to calculate a series of de-rived parameters (see below)

Path ShapeThis category describes the geometrical shape of the path

traveled by an animal Some of these parameters can be basedon unsigned degrees (absolute) and signed degrees (relative)

Heading The direction of movement in relation to a user-defined reference line

Turn angle Angle between the movement vectors of twoconsecutive sample intervals (absolute or relative)


Angular velocity Speed of change in direction of move-ment (ie amount of turning per unit of time [absolute or rela-tive]) The angular velocity of each sample is the turn angle forthat sample divided by the sample interval

Meander Change in direction of movement relative to thedistance moved (ie amount of turning per unit distance [ab-solute or relative] ) The mean of each sample is the turn anglefor that sample divided the distance moved from the last sample

Individual Behavioral StatesActivities of the animal are assigned to behavioral statesMovement Whether or not the tracked animalrsquos velocity ex-

ceeds a user-defined level The user can set thresholds for bothldquostart velocityrdquo and ldquostop velocityrdquo and the parameter is calcu-lated over a running average of a user-defined number of sam-ples The parameter has two states ldquomovingrdquo and ldquonot movingrdquo

Rearing The state in which a rodentrsquos body is verticallyerected It is a useful parameter for assessing exploratory be-havior It is detected by measuring the decrease in surface areawhen the rat or mouse stands up The user can define a runningaverage and percentage decrease in area to customize the param-eter for different species and conditions

Manually recorded behaviors While EthoVision is track-ing the animals the user can register any predefined behaviorsby pressing a key (or with the mouse) and these can be ana-lyzed in exactly the same way as any of the state variables auto-matically measured by EthoVision

Social InteractionsOn the basis of relative movement between pairs of simulta-

neously tracked animals distances and movements are classi-fied and assigned to particular parameters of social behavior

Distance between objects The distance between the centerof gravity of two animals

Proximity Whether or not the tracked animal is closer thana defined distance from another animal The user can definethreshold values for both ldquoin proximityrdquo and ldquonot in proximityrdquoto ensure that the frequency of transitions is biologically mean-ingful

Relative movement Whether or not an animal shows a rel-ative displacement toward (ldquomoving tordquo) or away from (ldquomov-ing fromrdquo) another tracked animal EthoVision does not justmeasure whether two objects get closer to or farther away fromeach other but the speed and direction of movement is takeninto account so that it is possible to correctly assess situationswhere Object A is moving toward Object B but B is movingaway from A (Spruijt et al 1992)

Speed of moving to and from The distance-weighted speedat which an animal moves toward or away from another animalCloser objects are given a heavier weighting and movementsbetween objects a long way from each other have a slower speedof movement ldquoSpeed of moving tordquo is calculated only if the rel-ative movement is ldquomoving tordquo

Net relative movement The signed distance-weightedchange in distance between two animals Net relative movementcombines the two parameters ldquoSpeed of moving fromrdquo andldquoSpeed of moving tordquo and can be used to quantify avoidance andaggression

The parameters of social interactions based on the relativemovement between pairs of objects can also be calculated forthe relative movement between animal(s) and zonespoints ofinterest (ie a static position in the arena) This allows the userfor example to calculate the speed with which an animal movestoward a novel object or when an animal is in proximity withthe center of the open field

(Manuscript received February 15 2001accepted for publication May 22 2001)

APPENDIX (Continued)


by the rat (Silverman Chang amp Russell 1988) Sensorscan also measure the position of the animal (and hencelocomotion) for instance by changes in the capacitanceof a plate when the animal is in proximity to it (ClarkeSmith amp Justesen 1992) or changes in body resistance(Tarpy amp Murcek 1984) Other comparable detectionmethods have included use of ultrasound (Akaka ampHouck 1980) and microwave Doppler radar (Martin ampUnwin 1980) A modern radar-based ldquoactometerrdquo isable to detect very small movements (which is particu-larly important in studies on insect behavior) and has theadvantage of working in complete darkness (Knoppienvan der Pers amp van Delden 2000) The position of a ratin an open field has been recorded by attaching the rat toa computer joystick via a series of rods attached by a col-lar to the ratrsquos neck (Brodkin amp Nash 1995) Animal be-havior can also be measured using a computerized ver-sion of a Skinner box (Skinner 1938) in which thesubject has to tap a touch-sensitive monitor (Morrison ampBrown 1990 Sahgal amp Steckler 1994) The motion ofindividual limbs can be monitored automatically usingactigraph sensors which detect movement by means ofa piezoelectric accelerometer (May et al 1996) An-other technique for automatic classification of behav-ioral patterns uses sensors to detect the mechanical vi-bration of a platform on which a cage with a mouse isplaced (Schlingmann van de Weerd Baumans Remieamp van Zutphen 1998)

Video TrackingVideo tracking systems were introduced in the early

1990s offering clear advantages of flexibility spatialprecision and accuracy over the various hardware deviceslisted above However with early systems the actual pathof the animal still had to be entered manually by the ex-perimenter by following the track of the animal with acomputer mouse (Pereira amp Oliveira 1994) a joystick(Morrel-Samuels amp Krauss 1990) or a digitizing tabletor similar device (Santucci 1995) Another early methodstill used in some commercially available systems is tofeed the analog video signal to a dedicated video track-ing unit which detects peaks in the voltage of the videosignal (indicating a region of high contrast between thetracked animal and the background) and use this to pro-duce the xy coordinates of the tracked animals This out-put is then fed to the serial port of a computer (KlapdorDulfer amp van der Staay 1996 Vorhees Acuff-SmithMinck amp Butcher 1992) These analog systems have thedisadvantage of being relatively inflexible (dedicated toparticular experimental setups) and can normally trackonly one animal in rather restricted lighting and back-ground conditions

Greater flexibility is achieved by the use of a videodigitizer The first video digitizers were of the columnscan type These had no internal memory of their ownand were able to sample the video signal only at ratherlow rates (Olivo amp Thompson 1988) In contrast a mod-ern frame grabber uses a high-speed analog-to-digital

converter to enable real-time conversion of the entirevideo image to a high-resolution grid of pixels It is alsopossible to acquire digitized video images by first con-verting the video input to a digital video format such asAVI and then using the AVI file as the input for objectdetection (Derry amp Elliot 1997) However this methodhas two disadvantages (1) The AVI file quickly getsvery large (and so only trials of a limited duration can becarried out) and (2) the method does not allow for real-time live data acquisition Alternatively a digital overlayboard can also be used to obtain positional data oftracked animals without the need for a frame grabber(Hartmann Assad Rasnow amp Bower 2000 RasnowAssad Hartmann amp Bower 1997)

A number of modern video tracking systems use framegrabbers to digitize analog video signals This enableshigh-speed data acquisition and therefore tracking ofanimals that are moving relatively fast However mostof these systems have severe limitations They can trackonly one animal in one arena If they can track multipleobjects they cannot identify them individually They tendto require simple backgrounds (in terms of their grayscale values) and can deal with only a limited range ofexperimental setups These systems also cannot handlecolor video

THE ETHOVISION SYSTEM

Development HistoryThe EthoVision video tracking system which we pre-

sent here was developed to overcome the limitations ofthe techniques and systems mentioned above EthoVisionhas been designed as a general-purpose video trackingmovement analysis and behavior recognition systemOn the basis of a high-resolution color video frame grab-ber and flexible software it is a versatile image process-ing system designed to automate behavioral observationand movement tracking on multiple animals simultane-ously against a variety of complex backgrounds

Our group has been involved with the developmentdeployment and support of EthoVision for almost a dec-ade A predecessor of the current EthoVision video track-ing system was first developed in the 1980s at the RudolfMagnus Institute for Neurosciences (RMI) of UtrechtUniversity (Spruijt Hol amp Rousseau 1992) In 1992Noldus Information Technology and the RMI joinedforces in order to develop this prototype into a maturevideo tracking system Over the years development pro-ceeded in close partnership with two universities (UtrechtUniversity and Wageningen University) and three phar-maceutical companies (Bayer AG Germany H Lund-beck AS Denmark Solvay Pharmaceuticals bv TheNetherlands) The first DOS-based implementation wasreleased in 1993 The system has undergone numerousupdates over the years on the basis of feedback fromusers around the world which has resulted in a compre-hensive package for studies of movement and behaviorThe software has recently been redesigned from the























































































DISCUSSION














REFERENCES


























































































































































































































DISCUSSION














REFERENCES
















































































































































REFERENCES




































































































































ethovision: a versatile video tracking system for automation of

Documents