virtual active touch using randomly patterned intracortical microstimulation

IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 1, JANUARY 2012 85

Virtual Active Touch Using Randomly PatternedIntracortical Microstimulation

Joseph E. O’Doherty, Mikhail A. Lebedev, Zheng Li, and Miguel A. L. Nicolelis

Abstract—Intracortical microstimulation (ICMS) has promiseas a means for delivering somatosensory feedback in neuropros-thetic systems. Various tactile sensations could be encoded bytemporal, spatial, or spatiotemporal patterns of ICMS. However,the applicability of temporal patterns of ICMS to artificial tactilesensation during active exploration is unknown, as is the minimumdiscriminable difference between temporally modulated ICMSpatterns. We trained rhesus monkeys in an active exploration taskin which they discriminated periodic pulse-trains of ICMS (200 Hzbursts at a 10 Hz secondary frequency) from pulse trains with thesame average pulse rate, but distorted periodicity (200 Hz burstsat a variable instantaneous secondary frequency). The statistics ofthe aperiodic pulse trains were drawn from a gamma distributionwith mean inter-burst intervals equal to those of the periodicpulse trains. The monkeys distinguished periodic pulse trainsfrom aperiodic pulse trains with coefficients of variation 0.25 orgreater. Reconstruction of movement kinematics, extracted fromthe activity of neuronal populations recorded in the sensorimotorcortex concurrent with the delivery of ICMS feedback, improvedwhen the recording intervals affected by ICMS artifacts wereremoved from analysis. These results add to the growing evidencethat temporally patterned ICMS can be used to simulate a tactilesense for neuroprosthetic devices.

Index Terms—Bidirectional interface, brain–machine interface,intracortical microstimulation, neural prosthesis.

I. INTRODUCTION

S ENSORY neuroprostheses and sensory substitution sys-tems for the restoration of hearing [1], [2] and vision

[3]–[8] have been investigated for several decades. Interestin neuroprosthetic devices that combine both motor and sen-sory components has developed more recently [9]–[14]. Oneexample of a bidirectional neuroprosthesis is a robotic limb

Manuscript received February 01, 2011; revised April 30, 2011, June16, 2011; accepted July 03, 2011. Date of publication December 27, 2011;date of current version January 25, 2012. This work was supported in partby the Defense Advanced Research Projects Agency (DARPA) under GrantN66001-06-C-2019, in part by the Telemedicine and Advanced Technology Re-search Center (TATRC) under Grant W81XWH-08-2-0119, and in part by theNational Institutes of Health through NICHD/OD under Grant RC1HD063390.The work of M. A. L. Nicolelis was supported by the NIH Director’s PioneerAward Program under Grant DP1OD006798.

J. E. O’Doherty is with the Department of Physiology and the W. M. KeckFoundation Center for Integrative Neuroscience, University of California, SanFrancisco, CA 94143 USA (e-mail: [email protected]).

M. A Lebedev and Z. Li are with the Department of Neurobiology andthe Center for Neuroengineering, Duke University, Durham, NC 27710 USA(e-mail: [email protected]; [email protected]).

M. A. L. Nicolelis is with the Departments of Neurobiology, Biomedical En-gineering, Psychology, and the Center for Neuroengineering, Duke University,Durham, NC 27710 USA (e-mail: [email protected]).

Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TNSRE.2011.2166807

controlled by brain activity while sensory information fromprosthetic sensors is delivered to somatosensory areas of thebrain [15], [16]. Other possible implementations include sen-sorized neuroprostheses for the restoration of bipedal walking[17] and putative systems combining both speech production[18], [19] and hearing [1], [2], [20].

In recent years, we have been studying intracortical micros-timulation (ICMS) delivered through microelectrode arrayschronically implanted in the primary somatosensory cortex(S1) as a means of adding a somatosensory feedback loopto a brain–machine interface (BMI) [9], [10], [21]. Takentogether with previous work showing that primates [22]–[24]and rodents [25]–[28] can discriminate ICMS patterns, there isgrowing evidence that ICMS of S1 could equip neuroprostheticlimbs with the sense of touch.

One of the neuroprosthetic devices that we envision in the fu-ture is a BMI-operated robotic arm that is equipped with touchsensors [9], [15], [16]. In such a sensorized neuroprosthesis, thetouch sensors would detect instances when the arm interactswith external objects sending signals to the brain in the formof ICMS. We have suggested that long-term operation of such asystem, which we call a brain–machine–brain interface (BMBI),could result in the incorporation of the prosthesis into the brain’srepresentation of the body, so that the artificial limb starts to actand feel as belonging to the subject [15]. Notwithstanding ini-tial encouraging results [9], [10], it is unclear whether ICMSwould be sufficient to reproduce the rich sensory information ofthe world of touch [29].

In particular, it is not well understood which kinds of ICMSpatterns are most useful for virtual active touch. Previously,we have shown that both New World [21] and Old Worldmonkeys [10] can discriminate temporal ICMS patterns ap-plied to S1 that consist of short (50–300 ms) high-frequency(100–400 Hz) pulse-trains presented at a lower secondaryfrequency (2–10 Hz). In these experiments, ICMS served asa cue that instructed the direction of reach. These patterns ofICMS could, in principle, mimic a wide variety of tactile inputs,especially when combined with spatial encoding [21]. Mod-ulations of sensory inputs in this frequency range correspondto the sensation of flutter [30]–[32]. These timescales are alsosimilar to neuronal modulations involved in texture encoding inthe somatosensory system [33]–[36], which makes such ICMSpatterns worthy candidates for exploration.

In this study, we examined the ability of rhesus monkeys todiscriminate a range of temporal ICMS patterns applied to S1 inthe context of an active exploration task in which ICMS mim-icked the tactile properties of virtual objects. We manipulatedthe ICMS patterns in a graded fashion, modulating the degreeof periodicity of the pulse-trains while maintaining a constant

1534-4320/$26.00 © 2011 IEEE

86 IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 1, JANUARY 2012

average pulse rate. We sought to determine the minimal pertur-bation of the periodic pattern that the monkeys could discrimi-nate. The degree of randomness (as quantified by the coefficientof variation, CV) was varied from trial to trial, which allowed usto quantify the monkeys’ sensitivity to ICMS frequency mod-ulations. Concurrently with ICMS delivery, we recorded fromlarge populations of cortical neurons using multielectrode im-plants. Kinematics of reach movements were extracted from thislarge-scale activity offline to estimate the accuracy of a BMBIwith a somatosensory feedback loop that transmits aperiodicICMS patterns.

This investigation of sensitivity to ICMS periodicity in S1was motivated by a possible application in neuroprostheticlimbs. We expect that the patterns of ICMS triggered by theinteraction of an upper-limb neuroprosthesis with objects in theenvironment could be highly irregular. The precise temporalstructure of such patterns would depend on the interactionof touch sensors in the robotic prosthesis with the specificsurface structure of the manipulated objects and on the specificexploratory movements used by the individual to interact withthe objects. Therefore, by knowing the limits of the nervoussystem in discriminating aperiodic ICMS patterns, we caninfer a principled upper bound on the maximum fidelity touchsensor that could be used in a neuroprosthesis, beyond whichno additional function would be restored.

This study builds on the results obtained by Romo et al. aboutICMS of S1 [22], [23], [31] and our own previous work [9], [21].One notable difference between the temporal ICMS patterns im-plemented here and those used by Romo et al. is that the aperi-odic patterns of ICMS that we used had the same mean pulse in-terpulse intervals as the periodic comparison ICMS pulse trains.Thus the average number of pulses in a pulse train was thesame for both periodic and aperiodic patterns. This allowed us toprobe S1 sensitivity to the temporal structure of ICMS withoutthe confound of average stimulus intensity. Romo et al. usedperiodic pulse trains with different frequencies [22], which leftopen the possibility that some of their results could be explainedby differences in average ICMS intensity.

Another major difference between this study and previousstudies of ICMS-evoked S1 sensations in primates is that theICMS patterns employed here were used in an active-explo-ration paradigm in which ICMS was used to simulate the tactileproperties of virtual objects. Our monkeys explored the virtualobjects, making self-paced exploratory movements, and decidedwhich objects to explore, in what order, and for how long. Thisis a more realistic model of a clinical somatosensory neuropros-thesis than previous designs.

II. METHODS

A. Implants

The experiments were conducted in two rhesus monkeys (Mand N) chronically implanted with multielectrode arrays in sev-eral cortical areas following our implantation methods [37]. Weused this same electrode array design for both large-scale neuralrecordings and ICMS delivery [9], [10]. Each monkey receivedfour 96-channel microelectrode arrays placed in the arm and leg

Fig. 1. Implants and task paradigm. (a) The monkeys were implanted withmicrowire arrays targeting M1 and S1 of the upper and lower limbs. (b)Channels used for stimulation with monkey M are accented in red. (c) Objectson the screen consisted of a central response zone surrounded by a peripheralfeedback zone. Movement of the avatar though the feedback or response zonetriggered the delivery of ICMS pulse trains. (d). Monkeys initiated a trialby holding the avatar in the center of the screen until two peripheral objectsappeared (500–1000 ms, random per trial; left sub-panel). Next, the monkeysfreely explored the objects (middle sub-panel). Finally, an object was selectedby holding the avatar within the response zone for 2000 ms (right sub-panel).

representation areas in sensorimotor cortex [Fig. 1(A)]. Eachhemisphere was implanted with two arrays: one in the arm rep-resentation and one in the leg representation. Within each array,electrodes were grouped in two 4 4 uniformly spaced grids ofelectrode triplets. The electrodes within each triplet had dif-ferent lengths, staggered at 300 m intervals. One grid wasaligned over primary motor cortex (M1) and the other over S1.The monkeys were implanted for a series of studies beyondthose described here. For the purpose of this study, we recordedneuronal activity from the right hemisphere arm arrays while themonkeys performed a manual task with their left hands. Stim-ulation was applied to the right hemisphere arm subdivision ofS1 in monkey M and the right hemisphere leg subdivision of S1in monkey N. All animal procedures were performed in accor-dance with the National Research Council’s Guide for the Careand Use of Laboratory Animals and were approved by the DukeUniversity Institutional Animal Care and Use Committee.

B. Behavioral Task

The monkeys were trained in a reaching task in which theymanipulated a hand-held joystick to move a virtual reality arm(avatar) displayed on a computer screen [Fig. 1(C) and (D)].The monkeys reached with the avatar arm towards screen ob-jects and searched for an object with a particular artificial tex-ture indicated by ICMS of S1. The objects were circular in shapeand visually identical. Each monkey was previously trained inother variants of this task. In this study, the monkeys were showntwo objects, one of which was associated with a periodic ICMSpattern and the other with an aperiodic pattern [Fig. 2(A)]. Themonkeys were rewarded for selecting the object paired withperiodic ICMS. The objects appeared at different locations onthe screen, with the constraint that the distance from the screencenter to each object was fixed, and the angle between the ob-jects was 180 (i.e., centrally symmetric). If the correct object

O’DOHERTY et al.: VIRTUAL ACTIVE TOUCH USING RANDOMLY PATTERNED INTRACORTICAL MICROSTIMULATION 87

Fig. 2. Example aperiodic ICMS pulse trains. (a) Raster indicates the range ofvariability of inter-burst intervals (�� , cyan; �� , blue; ��

��, green; �� , red; �� , black). Note that each vertical lineindicates a short burst of ICMS, not a single pulse. (a) Distributions of inter-burstintervals corresponding with the rasters shown in (a).

was selected, the monkeys were rewarded with a drop of fruitjuice.

Each trial commenced when the monkey grasped the joystickwith its left hand. At this point, a circular target appeared in thecenter of the screen. The monkey placed the avatar arm on thatcenter target for 0.5–1.0 s. Then, the central target disappearedand two peripheral objects appeared. Each consisted of a centralresponse zone and a peripheral feedback zone [Fig. 1(C)]. Whenthe avatar hand entered the feedback or response zones, ICMSpulse trains were delivered to S1. A trial was concluded witha reward when the monkey placed the avatar hand within theresponse zone of the correct object for 2 s; no reward was deliv-ered if the incorrect object was selected. The monkeys were per-mitted to explore the virtual objects in any sequence, but the trialended when they stayed over an object’s response zone longerthan the hold period of 2 s. Then a 0.5 s delay was issued beforethe next trial began.

C. ICMS Patterns

ICMS trains consisted of symmetric [38], biphasic, charge-balanced pulses of ICMS delivered in a bipolar fashion throughadjacent pairs of microwires [9], [21]. For monkey M, the an-odic and cathodic phases of stimulation each had amplitudes of150 A and pulse widths of 105 s; for monkey N, 150 A and200 s, respectively. The anodic and cathodic phases were sep-arated by 25 s.

ICMS was delivered to different subdivisions of S1 for eachmonkey. For monkey M, the hand representation area of S1 wasused as the target for ICMS, so that the monkey experiencedputative sensations in its hand [Fig. 1(B)]. For monkey N, ICMSwas applied to the thigh representation area of S1. Two electrodepairs were used for each animal.

The temporal pattern of ICMS consisted of 200 Hz pulsetrains delivered for 50 ms and presented at a lower secondaryfrequency. The secondary frequency for the rewarded artificialtexture was a constant 10 Hz. For the unrewarded textures, thetiming of the ICMS bursts was aperiodic. The interval betweeneach aperiodic burst was a random variable drawn from agamma distribution of instantaneous inter-burst intervals [39]

(1)

where is the probability density function, is the inter-burstinterval, is the shape parameter, is the scale parameter, and

is the Gamma function. We computed the shape and scale pa-rameters as a function of the mean inter-pulse interval, , andthe coefficient of variation, CV, the ratio of the standard devia-tion to the mean

(2)

This allowed the construction of aperiodic pulse trains withinter-burst intervals equal in expectation to the periodic pulsetrains while giving control over the degree of aperiodicity: thehigher the CV, the more aperiodic the pulse-train. A pulse trainwith a CV of zero was equivalent to the periodic, rewardedpattern. The average number of ICMS pulses per unit time wasthe same for the periodic and aperiodic patterns. Examples ofpulse trains with different CVs are shown in Fig. 2.

D. Artifact Suppression

An important question arising from the use of ICMS for sen-sory feedback is whether the stimulation causes artifacts in cor-tical neural ensemble recordings and how these artifacts canbe dealt with to minimize their impact on BMI operations. Toaddress this question, we processed the neural recordings byremoving (blanking) a window of neural activity immediatelysubsequent to each pulse of ICMS and then performed decodingwith the processed data. By systematically varying the length ofthe blanking intervals we could determine the amount of artifactremoval that produced the most accurate movement reconstruc-tions.

Artifact removal was implemented as follows. The stim-ulation artifacts had stereotypical shapes when recorded bythe spike acquisition system, and we could reliably detectartifacts by using spike-sorting templates that matched theartifact shapes, allowing us to determine the precise time ofeach stimulation pulse. We then ignored all spiking on everychannel for milliseconds after each stimulation pulse, where

varied in value from 0 to 10, in integer steps. This was doneby first counting spikes in 1 ms nonoverlapping time windows(i.e., binning at 1 ms resolution). We then zeroed the spikecounts in bins that were equal to or less than ms after thestimulation pulse. For example, for , we zeroed the binsat , and , where is the 1 ms bin of thestimulation pulse [see Fig. 6(A)]. For , we zeroed thebin at the stimulation pulse only. Then, we summed adjacentbins to produce spike counts in 100 ms nonoverlapping bins forour decoders. For this last step, we adjusted the spike counts


Fig. 3. Summary of the behavioral performance of monkeys M (circles) andN (diamonds) for 10 sessions as they learned the task. Each symbol shows themean performance for the session. Filled symbols depict sessions with perfor-mance significantly different from chance (chance level of 50%; � � ��,one-sided binomial test). Curves are the sigmoidal lines of best fit.

by multiplying by the quantity , whereis the number of zeroed 1 ms bins in the 100 ms bin.

This operation preserves, in expectation, the number of spikesin each 100 ms bin, by performing extrapolation.

E. Kinematics Extraction

The X and Y position of the avatar was extracted from cor-tical activity using a fifth-order unscented Kalman filter [40] andWiener filter [37]. For both algorithms, we evaluated the de-coding accuracy after artifact blanking. We performed two-foldcross-validation with each algorithm on 26 sessions, 13 fromeach monkey. For the unscented Kalman filter, we used a tuningmodel with linear weights for position, velocity, distance fromcenter of workspace, and magnitude of velocity. The unscentedKalman filter had three future taps, two past taps, and one tapin the movement model (see [40] for details). The tuning modelweights were fit with adaptive ridge regression [41], with theridge parameter found by cross-validation on the training data.For the Wiener filter, we used 10 taps of spiking history and pre-dicted the position only. The Wiener coefficients were fit usingridge regression with the ridge parameter found by cross-vali-dation on the training data.

III. RESULTS

A. Learning

Initially, both monkeys were required to discriminate be-tween the periodic (rewarded) ICMS pattern and an aperiodicpattern with a CV of 0.8. Each monkey learned this discrim-ination task in approximately eight daily sessions (Fig. 3).Monkey N stabilized at a performance level of approximately90% correctly executed trials, monkey M at an 85% level.These learning curves are consistent with our previous resultson rhesus monkey learning with ICMS-instructed tasks [9].

Fig. 4. Psychometric curves for different coefficients of variation on the aperi-odic pulse trains. (a) Mean performance at differentiating periodic versus aperi-odic ICMS pulse trains as a function of CV for monkey N. Each symbol repre-sents the mean performance across sessions; error bars indicate 95% confidenceintervals. Curves are the sigmoidal lines of best fit. Symbols are as in Fig. 3. (b)Same as (a), but for monkey M.

B. Psychometrics

After both monkeys learned to discriminate periodic ICMSfrom aperiodic with a CV of 0.8, we began to vary the CV ofthe aperiodic ICMS pattern on every trial. In these sessions, thedistribution of CVs was picked so that for half of the trials theCV of the unrewarded object was greater or equal to 0.6. Thesesessions continued for two weeks, yielding a database for psy-chometric analysis.

Psychometric curves (i.e., graphs showing the proportion ofcorrectly performed trials as the function of CV, Fig. 4) indi-cated a clear dependency of discrimination accuracy on the de-gree of randomness of the comparison ICMS pattern. Perfor-mance stabilized for CVs higher than 0.8 and gradually de-creased for CVs lower than that value. The threshold CV fordiscrimination for both monkeys was 0.25. Below this value,the monkeys performed at chance levels.

C. Active Exploration

Discrimination of ICMS patterns was performed through ac-tive exploration: a monkey would probe the feedback zone ofan object with the avatar to acquire an ICMS pattern and theneither select that object if it perceived the ICMS pattern as peri-odic or explore the other object if the pattern was judged as dif-ferent from periodic. This active exploration was evident froman analysis of object exploration intervals [Fig. 5(A) and (B)].We designated intervals during which the avatar hand continu-ously stayed over a given object as “visits.” For very low CVs,the statistics of visit durations were the same for periodic andaperiodic patterns [Fig. 5(A)].

The distribution of these intervals indicated short (less than2 s) exploratory visits and a prominent peak at 2 s that corre-


Fig. 5. Active exploration of the virtual objects as a function of CV. (a) His-tograms of visit durations to the rewarded object (�� , black trace) versusthe unrewarded object (�� , red trace) for trials with that CV combina-tion. Visits are quantified as intervals when the avatar was continuously over thefeedback zone or the response zone. Vertical line indicates the hold interval (2 s).(b) Same as (a), but for trials with unrewarded objects with aperiodic ICMS CVsof 0.8 (cyan trace). (c) Fraction of the short visits (less than 2 s) for the rewardedobject (square symbols) versus the unrewarded object (triangular symbols) ex-pressed as a function of CV. Gray shaded zones correspond to the data in (a)and (b).

sponded to selecting an object. This is because the monkey wasrequired to hold the avatar hand over the response zone of theobject for 2 s to obtain a reward (or to get a trial cancellation ifthe object was selected incorrectly). Accordingly, the peak at 2 scorresponded to visits for which the monkey selected a given ob-ject. Intervals longer than 2s were possible because visits com-prised the portion spent over the feedback zone (but outside ofthe response zone) as well as the time spent over the responsezone. We called visits with durations less than 2 s short visits,and those with durations of 2 s or longer long visits. Both theshort-visit portion of the distribution and the long-visit part werepreserved for periodic ICMS ( , Fig. 5(A), black line)versus weakly aperiodic ICMS e.g., CV of 0.05 (Fig. 5(A), redline).

The distributions of visit-durations were markedly differentfor higher CVs [Fig. 5(B)]. The distribution of visit-durationsfor aperiodic ICMS with a CV of 0.8 (cyan line) revealed a pre-dominance of short visits with an average duration of 0.8 s anda small proportion of long visits. For the periodic pattern (blackline), the distribution showed the predominance of long visits.These data indicate that it took the monkey on average 0.8 s torecognize the unrewarded aperiodic ICMS pattern and to switchto the correct object (periodic pattern) when sufficiently aperi-odic ICMS patterns were used.

The change in monkey exploratory behavior for different de-grees of ICMS-pattern aperiodicity is clear from the statistics ofvisits, expressed as the proportion of short visits normalized bythe total number of visits [Fig. 5(C)]. When a monkey touched

Fig. 6. Exploration of blanking intervals. (a) Schematic of the blanking proce-dure. Spikes and ICMS pulses were categorized into 1 ms bins. For each bin con-taining an ICMS pulse, that bin and a variable number of bins (three shown here)were blanked subsequently. (b). Mean movement reconstruction accuracy as afunction of blanking intervals for both monkeys (symbols as in Fig. 3) and bothalgorithms (Wiener filter, WF, dashed lines; unscented Kalman filter, UKF, solidlines). Bars indicate standard error. Shaded region corresponds to no blanking.

an object associated with an aperiodic pattern (Fig. 5(C), trian-gles), it tended to make more short visits than when the monkeytouched an object associated with a periodic pattern (squares).For high CVs (CV greater than 0.7), the proportion of shortvisits constituted approximately 80% of the total number ofvisits. For lower CVs, this value decreased, indicating that themonkey made a decision to stay on the unrewarded object moreoften.

D. Kinematics Extraction

Fig. 6 shows the average accuracy for the extraction of avatarposition from cortical ensemble activity for different lengths ofartifact blanking intervals. Consistent with our previous results[40], the unscented Kalman filter consistently outperformed theWiener filter. For monkey M, the peak accuracy was

dB (mean standard error) for the unscented Kalman filterand dB for the Wiener filter. For monkey N, thesevalues were dB and dB, respectively. Theseaccuracy values are within the range that we typically observefor BMI predictions [9], [17], [41].


Both algorithms benefited somewhat from artifact blanking,more so for monkey M. For monkey M, maximum accuracy wasachieved with 5 ms of artifact blanking for both the Weiner filterand the unscented Kalman filter. For monkey N, maximum ac-curacy was achieved with 2 ms of artifact blanking for both de-coders. These values reflect the difference in artifact durationand amplitude in two monkeys. The artifacts were more promi-nent and of longer duration in monkey M because of the closeproximity of the stimulation site (hand representation of S1) tothe area where neuronal activity was collected (arm represen-tation of M1 and S1). The artifacts were smaller and of shorterduration for monkey N, which received stimulation in the legrepresentation area of S1 with recordings performed in the armrepresentation area.

Curiously, the performance of the unscented Kalman filterwas slightly better for the no blanking condition than for 0 msof blanking. This was because the recording channels that de-tected ICMS artifacts occasionally recorded additional mechan-ical artifacts related to monkey head movements. Apparently,the filter could utilize these mechanical artifacts that influencedthe spike recording channels (blanked them or introduced erro-neous spikes) to improve predictions, and its performance wasvery slightly reduced when these artifacts were removed. Thisunderscores the importance of registering the artifacts and re-moving them to minimize their influence on the filter perfor-mance.

To quantify the decrease in predictions caused by the presenceof ICMS artifacts, we recorded from both monkeys as they per-formed a center-out task without any ICMS. For this task, theyhad to move the avatar from the center of the screen to a singleperipheral object and hold for 2 s. For monkey M, accuracy inthis task was 23% higher than during the ICMS sessions with theunscented Kalman filter and 32% higher with the Wiener filter.For monkey N, these values were 2.7% and 27%, respectively.Thus, the artifacts worsened the predictions, even after optimalblanking, but still within a tolerable range.

IV. DISCUSSION

This study continued our work on the development of an arti-ficial somatosensory channel for BMIs [9], [10], [21]. Monkeysscanned virtual objects with an avatar hand and discriminatedtheir artificial textures as represented by temporal patterns ofICMS. This paradigm models the requirements of a clinicallyrelevant neuroprosthetic arm sensorized with an artificial tactilechannel. Such a neuroprosthetic arm could be used to touch ex-ternal objects and estimate their tactile properties (roughness/smoothness, hardness/softness, wetness/dryness, temperature)using sensors on the prosthetic hand. The transmission of thisinformation to the nervous system is a difficult problem becauseof the artificial nature of the stimulation methods. We exploredthe capability of temporally patterned ICMS as a way to deliversomatosensory feedback to the brain by parametrically varyingthe degree of randomness of ICMS trains. Monkeys learnedto distinguish regular ICMS patterns from irregular ones, a re-sult which suggests that they were able to discriminate the finetemporal structure of ICMS trains. Irregular bursts of sensorydischarges are expected to occur in practical neuroprostheses,when the prosthesis interacts with realistically textured objects.

A neuroprosthetic hand used to scan a ridged surface, for ex-ample, would generate an ICMS burst each time a ridge inter-acts with the prosthetic sensor. In this setting, the degree of pe-riodicity of ICMS pulse trains could inform the prosthesis userabout the regularities or irregularities of an object’s material orshape.

Our results complement previous work on ICMS frequencydiscrimination conducted by Romo et al. who trained their mon-keys to discriminate periodic ICMS pulse trains [22]–[24] and todiscriminate the mean rate of aperiodic pulse trains [22], [42].Our study expanded the range of temporal patterns that couldbe represented by ICMS of S1 by changing the regularity ofthe secondary frequency. Moreover, ICMS in our experimentsserved as somatosensory feedback during virtual active touch,rather than merely a cue in a forced choice task as in the ma-jority of previous studies. The animals actively explored vir-tual objects with an avatar hand, spending similar times overthese objects as would be needed for normal interaction withthe environment. Additionally, chronically implanted electrodeswere used for ICMS delivery, which allowed us to monitor long-term learning to utilize ICMS as sensory feedback. In previousstudies, stimulating electrodes were often inserted in the brainanew during each daily session. Long-term usage of ICMS inthe present experiments (as well as in previous experimentswith the same monkeys) did not result in deterioration of per-formance, which indicates that the charge-balanced ICMS usedhere did not damage the electrodes or brain tissue, or that anysuch damage was below a threshold where it would begin to im-pact task performance.

Our results show that monkeys detect distortions in the 10 HzICMS secondary frequency after random variations of that fre-quency exceeded 25%, that is, instantaneous frequency fluctu-ated from 7.5 to 12.5 Hz. This estimate of the detection thresholdcan be used in future neuroprosthetic designs as a characteristicsensitivity value. Future studies should probe the sensitivity ofdiscrimination to different primary and secondary frequencies.Additionally, spatiotemporal ICMS [21] and ICMS of differentdurations should be explored as ways to encode information inBMBI sensory channels.

The interaction of a neuroprosthesis with realistically tex-tured objects in the natural world will inevitably result in astream of temporally patterned sensory information. Either theuser or the neuroprosthesis (or some combination thereof) willtherefore need to deal with these signals. Texture analysis couldbe delegated, in part, to a shared control algorithm [43]. In thismode of operation, a sensation processor would analyze raw sig-nals from sensors on the prosthesis and interpret them in thecontext of how the neuroprosthetic device “skin” moved againstthe surface of textured objects. Simplified ICMS patterns—rep-resenting different classes of textures—could then be sent tothe brain. Alternatively, ICMS could directly encode a signalrepresenting both the spatiotemporal movements of the pros-thetic limb and the intrinsic microstructure of the material beingtouched. In this case, the temporal patterns of ICMS would haveto be interpreted by the user in the context of the particular ex-ploration pattern used [44]. The choice of the encoding schemewill likely be dictated by the requirements of the specific neu-roprosthetic application.


It would be of interest for future studies to explore the optimaltemporal properties of ICMS modulations at different S1 sites.Romo et al. reported best results when they applied ICMS torapidly adapting neurons in area 3b [22]. We stimulated in area1, where the distinction between rapidly adapting and slowlyadapting categories of neurons is less clear. Additionally, weused multi-session training periods with chronically implantedelectrodes, in contrast to Romo et al., who used independentstimulation sessions with acute electrodes. It is possible thatthat our longer training interval facilitated the discrimination ca-pacity of the monkeys. The distinction between different S1 lo-cations and the role of learning will need to be studied in moredetail in future studies.

BMBIs equipped with afferent ICMS feedback loops needto compensate for electrical artifacts produced by ICMS pulsesthat may interfere with neuronal recordings. In our previousBMBI designs, we either discounted the entire period of ICMSapplication [9] or used interleaved recording and ICMS deliveryintervals [10]. These previous approaches limited the flexibilityof ICMS delivery. In this study, we did not impose limitationson the timing of ICMS delivery and treated ICMS artifacts asthey occurred. We found that blanking the periods after ICMSdelivery by short intervals (2–5 ms) improved the accuracy ofextraction of limb kinematics from neuronal activity. Overallaccuracy of predictions was 20%–30% less as compared to ses-sions in which ICMS was not used. Nonetheless, the predic-tions were still acceptable and within range of previously re-ported accuracy of BMI decoding. This result suggests that arti-fact blanking is practical for bidirectional neuroprostheses usingirregular ICMS pulse trains.

The precise character of perceptions evoked by periodicversus aperiodic patterns of ICMS will have be evaluated inhuman subjects [45]. There is a suggestion by Fridman et al.that ICMS amplitude, pulse-width, and frequency all interact tocontribute to a unitary perception of “perceived intensity” [46].Therefore, one might argue that our monkeys discriminatedthe periodic and aperiodic pulse trains on the basis of their in-stantaneous peak intensities rather than their temporal patterns.However, this simple explanation is unlikely because the peakinstantaneous frequency of ICMS was 200 Hz for both theperiodic as well as the aperiodic artificial textures. Therefore,our results indicate that the monkeys must have been using astrategy beyond simply detecting the maximum instantaneousfrequency. One possible neural implementation could employa leaky integrator mechanism that detected variability of theICMS secondary frequency by integrating neural responses toICMS within an optimal time window, thus detecting transientincreases in ICMS frequency. This and other alternative mech-anisms will need to be elucidated with future studies.

Our current and previous [9], [10], [21] results suggest thatnew perceptions may evolve as subjects practice with ICMS.We observed that it took monkeys 1–2 weeks to start to under-stand ICMS, even if they were previously overtrained with a vi-brotactile variant in the same task. However, once they learnedthe first ICMS task, learning subsequent tasks took much lesstime. A virtual active touch setting where subjects evoke ICMSand associated sensations through their own actions [47]–[49]may contribute to shaping the artificial perception and lead to

the development of anticipatory cortical modulations similar tocorollary discharge [50].

The problem of artifacts will be compounded as multiplestimulation channels are employed with asynchronously deliv-ered pulses. Excessive masking of the recordings by ICMS ar-tifacts should be avoided in the design of such systems. In thefuture, the problem of artifacts [51], as well as the unreliablespatial extent of ICMS [52] could be mitigated by optogeneticstimulation [53]–[55].

ACKNOWLEDGMENT

The authors would like to thank D. Dimitrov for assistancewith the animal surgeries, S. Shokur for design and program-ming of the monkey avatar, and G. Lehew, J. Meloy, T. Phillips,L. Oliveira, and S. Halkiotis for invaluable technical support.

REFERENCES

[1] M. M. Merzenich, D. N. Schindler, and M. W. White, “Feasibility ofmultichannel scala tympani stimulation,” Laryngoscope, vol. 84, pp.1887–1893, Nov. 1974.

[2] J. B. Fallon, D. R. F. Irvine, and R. K. Shepherd, “Cochlear implantsand brain plasticity,” Hear. Res., vol. 238, pp. 110–117, Apr. 2008.

[3] P. Bach-y-Rita, C. C. Colins, F. A. Saunders, B. White, and L. Scadden,“Vision substitution by tactile image projection,” Nature, vol. 221, pp.963–964, Mar. 1969.

[4] P. Bach-y-Rita, K. A. Kaczmarek, M. E. Tyler, and J. Garcia-Lara,“Form perception with a 49-point electrotactile stimulus array on thetongue: A technical note,” J. Rehabil. Res. Dev., vol. 35, pp. 427–430,Oct. 1998.

[5] P. Bach-y-Rita and S. W. Kercel, “Sensory substitution and the human-machine interface,” Trends Cogn. Sci., vol. 7, pp. 541–546, Dec. 2003.

[6] W. H. Dobelle, M. G. Mladejovsky, and J. P. Girvin, “Artificial visionfor the blind: Electrical stimulation of visual cortex offers hope for afunctional prosthesis,” Science, vol. 183, pp. 440–444, Feb. 1974.

[7] G. Dagnelie, “Psychophysical evaluation for visual prosthesis,” Annu.Rev. Biomed. Eng., vol. 10, pp. 339–368, 2008.

[8] E. D. Cohen, “Prosthetic interfaces with the visual system: Biologicalissues,” J. Neural Eng., vol. 4, pp. R14–R31, Jun. 2007.

[9] J. E. O’Doherty, M. A. Lebedev, T. L. Hanson, N. A. Fitzsimmons,and M. A. L. Nicolelis, “A brain-machine interface instructed by directintracortical microstimulation,” Front. Integr. Neurosci., vol. 3, p. 20,2009.

[10] J. E. O’Doherty et al., “Active tactile exploration enabled by a brain-machine-brain interface,” Nature, vol. 479, pp. 228–231, Nov. 2011.

[11] F. A. Mussa-Ivaldi et al., “New perspectives on the dialogue betweenbrains and machines,” Front. Neurosci., vol. 4, p. 44, 2010.

[12] S. Stanslaski et al., “An implantable bi-directional brain-machine inter-face system for chronic neuroprosthesis research,” in Proc. IEEE Eng.Med. Biol. Soc. Conf., 2009, vol. 2009, pp. 5494–5497.

[13] A. H. Fagg et al., “Toward a biomimetic, bidirectional, brain machineinterface,” in Proc. IEEE Eng. Med. Biol. Soc. Conf., 2009, vol. 2009,pp. 3376–3380.

[14] T. C. Marzullo, M. J. Lehmkuhle, G. J. Gage, and D. R. Kipke, “De-velopment of closed-loop neural interface technology in a rat model:Combining motor cortex operant conditioning with visual cortex mi-crostimulation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 18, no.4, pp. 117–126, Apr. 2010.

[15] M. A. Lebedev and M. A. L. Nicolelis, “Brain-machine interfaces: Past,present and future,” Trends Neurosci., vol. 29, pp. 536–546, Sep. 2006.

[16] M. A. L. Nicolelis and M. A. Lebedev, “Principles of neural ensemblephysiology underlying the operation of brain-machine interfaces,” Nat.Rev. Neurosci., vol. 10, pp. 530–540, Jul. 2009.

[17] N. A. Fitzsimmons, M. A. Lebedev, I. D. Peikon, and M. A. L.Nicolelis, “Extracting kinematic parameters for monkey bipedalwalking from cortical neuronal ensemble activity,” Front. Integr.Neurosci., vol. 3, p. 3, 2009.

[18] J. S. Brumberg, A. Nieto-Castanon, P. R. Kennedy, and F. H. Guen-ther, “Brain-computer interfaces for speech communication,” SpeechCommun., vol. 52, pp. 367–379, Apr. 2010.


[19] F. H. Guenther et al., “A wireless brain-machine interface for real-timespeech synthesis,” PLoS One, vol. 4, p. e8218, 2009.

[20] N. R. Peterson, D. B. Pisoni, and R. T. Miyamoto, “Cochlear implantsand spoken language processing abilities: Review and assessment ofthe literature,” Restor. Neurol. Neurosci., vol. 28, pp. 237–250, 2010.

[21] N. A. Fitzsimmons, W. Drake, T. L. Hanson, M. A. Lebedev, and M. A.L. Nicolelis, “Primate reaching cued by multichannel spatiotemporalcortical microstimulation,” J. Neurosci., vol. 27, pp. 5593–5602, May2007.

[22] R. Romo, A. Hernández, A. Zainos, and E. Salinas, “Somatosensorydiscrimination based on cortical microstimulation,” Nature, vol. 392,pp. 387–390, Mar. 1998.

[23] R. Romo, A. Hernández, A. Zainos, C. D. Brody, and L. Lemus,“Sensing without touching: Psychophysical performance based oncortical microstimulation,” Neuron, vol. 26, pp. 273–278, Apr. 2000.

[24] V. de Lafuente and R. Romo, “Neuronal correlates of subjective sen-sory experience,” Nat. Neurosci., vol. 8, pp. 1698–1703, Dec. 2005.

[25] S. Butovas and C. Schwarz, “Detection psychophysics of intracorticalmicrostimulation in rat primary somatosensory cortex,” Eur. J. Neu-rosci., vol. 25, pp. 2161–2169, Apr. 2007.

[26] A. R. Houweling and M. Brecht, “Behavioural report of single neuronstimulation in somatosensory cortex,” Nature, vol. 451, pp. 65–68, Jan.2008.

[27] S. K. Talwar et al., “Rat navigation guided by remote control,” Nature,vol. 417, pp. 37–38, May 2002.

[28] S. Venkatraman and J. M. Carmena, “Active sensing of target locationencoded by cortical microstimulation,” IEEE Trans. Neural Syst. Re-habil. Eng., vol. 19, no. 3, pp. 317–324, Jun. 2011.

[29] D. Katz and L. E. Krueger, The World of Touch. Hillsdale, N.J.: Erl-baum, 1989.

[30] V. B. Mountcastle, M. A. Steinmetz, and R. Romo, “Frequency dis-crimination in the sense of flutter: Psychophysical measurements cor-related with postcentral events in behaving monkeys,” J. Neurosci., vol.10, pp. 3032–3044, Sep. 1990.

[31] E. Salinas, A. Hernández, A. Zainos, and R. Romo, “Periodicity andfiring rate as candidate neural codes for the frequency of vibrotactilestimuli,” J. Neurosci., vol. 20, pp. 5503–5515, Jul. 2000.

[32] M. A. Lebedev, J. M. Denton, and R. J. Nelson, “Vibration-entrainedand premovement activity in monkey primary somatosensory cortex,”J. Neurophysiol., vol. 72, pp. 1654–1673, Oct. 1994.

[33] R. J. Sinclair and H. Burton, “Tactile discrimination of gratings: Psy-chophysical and neural correlates in human and monkey,” Somatosens.Mot. Res., vol. 8, pp. 241–248, 1991.

[34] R. J. Sinclair, J. R. Pruett, and H. Burton, “Responses in primary so-matosensory cortex of rhesus monkey to controlled application of em-bossed grating and bar patterns,” Somatosens. Mot. Res., vol. 13, pp.287–306, 1996.

[35] E. Gamzu and E. Ahissar, “Importance of temporal cues for tactile spa-tial- frequency discrimination,” J. Neurosci., vol. 21, pp. 7416–7427,Sept. 2001.

[36] J. R. Phillips and K. O. Johnson, “Tactile spatial resolution. II. Neuralrepresentation of bars, edges, and gratings in monkey primary affer-ents,” J. Neurophysiol., vol. 46, pp. 1192–1203, Dec. 1981.

[37] J. M. Carmena et al., “Learning to control a brain-machine interface forreaching and grasping by primates,” PLoS Biol., vol. 1, p. E42, Nov.2003.

[38] A. Koivuniemi and K. Otto, “Asymmetric vs. symmetric electric pulsesfor intracortical microstimulation,” IEEE Trans. Neural Syst. Rehabil.Eng., vol. 19, no. 5, pp. 468–476, Oct. 2011.

[39] A. D. Dorval, A. M. Kuncel, M. J. Birdno, D. A. Turner, and W. M.Grill, “Deep brain stimulation alleviates parkinsonian bradykinesia byregularizing pallidal activity,” J. Neurophysiol., vol. 104, pp. 911–921,Aug. 2010.

[40] Z. Li et al., “Unscented Kalman filter for brain-machine interfaces,”PLoS One, vol. 4, p. e6243, 2009.

[41] Y. Grandvalet, , L. Niklasson, Ed. et al., “Perspectives in Neural Com-puting,” in Least Absolute Shrinkage is Equivalent to Quadratic Penal-ization. New York: Springer Verlag, 1998, pp. 201–206.

[42] A. Hernández, A. Zainos, and R. Romo, “Neuronal correlates of sen-sory discrimination in the somatosensory cortex,” in Proc. Nat. Acad.Sci. USA, May 2000, vol. 97, pp. 6191–6196.

[43] H. K. Kim et al., “Continuous shared control for stabilizing reachingand grasping with brain-machine interfaces,” IEEE Trans. Biomed.Eng., vol. 53, no. 6, pp. 1164–1173, Jun. 2006.

[44] S. J. Lederman and R. L. Klatzky, “Hand movements: A window intohaptic object recognition,” Cogn. Psychol., vol. 19, pp. 342–368, Jul.1987.

[45] E. Heming, R. Choo, J. Davies, and Z. Kiss, “Designing a thalamicsomatosensory neural prosthesis: Consistency and persistence of per-cepts evoked by electrical stimulation,” IEEE Trans. Neural Syst. Re-habil. Eng., vol. 19, no. 5, pp. 477–482, Oct. 2011.

[46] G. Y. Fridman, H. T. Blair, A. P. Blaisdell, and J. W. Judy, “Perceivedintensity of somatosensory cortical electrical stimulation,” Exp. BrainRes., vol. 203, pp. 499–515, June 2010.

[47] R. Sinclair and H. Burton, “Responses from area 3b of somatosensorycortex to textured surfaces during active touch in primate,” Somatosens.Res., vol. 5, pp. 283–310, 1988.

[48] C. Simões-Franklin, T. A. Whitaker, and F. N. Newell, “Active andpassive touch differentially activate somatosensory cortex in textureperception,” Hum. Brain Mapp., vol. 32, pp. 1067–1080, 2011.

[49] S. J. Bolanowski, R. T. Verrillo, and F. McGlone, “Passive, active andintra-active (self) touch,” Behav. Brain. Res., vol. 148, pp. 41–45, Jan.5, 2004.

[50] T. B. Crapse and M. A. Sommer, “Corollary discharge across the an-imal kingdom,” Nat. Rev. Neurosci., vol. 9, pp. 587–600, Aug. 2008.

[51] J. D. Rolston, R. E. Gross, and S. M. Potter, “A low-cost multielectrodesystem for data acquisition enabling real-time closed-loop processingwith rapid recovery from stimulation artifacts,” Front. Neuroeng., vol.2, p. 12, 2009.

[52] M. H. Histed, V. Bonin, and R. C. Reid, “Direct activation of sparse,distributed populations of cortical neurons by electrical microstimula-tion,” Neuron, vol. 63, pp. 508–522, Aug. 2009.

[53] F. Zhang, A. M. Aravanis, A. Adamantidis, L. de Lecea, and K. Deis-seroth, “Circuit-breakers: Optical technologies for probing neural sig-nals and systems,” Nat. Rev. Neurosci., vol. 8, pp. 577–581, Aug. 2007.

[54] E. S. Boyden, F. Zhang, E. Bamberg, G. Nagel, and K. Deisseroth,“Millisecond-timescale, genetically targeted optical control of neuralactivity,” Nat. Neurosci., vol. 8, pp. 1263–1268, Sep. 2005.

[55] F. Zhang, L. P. Wang, E. S. Boyden, and K. Deisseroth, “Channel-rhodopsin-2 and optical control of excitable cells,” Nat. Methods, vol.3, pp. 785–792, Oct. 2006.

Joseph E. O’Doherty received the B.S. degree inphysics from East Carolina University, Greenville,NC, in 2001 and the Ph.D. degree in biomedicalengineering from Duke University, Durham, NC, in2011.

He is currently a Postdoctoral Scholar at the W.M. Keck Foundation Center for Integrative Neuro-science, in San Francisco, CA. He has held a previousresearch appointment at the Duke University Centerfor Neuroengineering, Durham, NC (2011). His re-search interests include methods for providing artifi-

cial somatic sensation and proprioception for neural prostheses.

Mikhail A. Lebedev received the M.S. degreein physics from the Moscow Institute of Physicsand Technology, Moscow, Russia, in 1986 and thePh.D. degree in neurobiology from the University ofTennessee, Memphis, in 1995.

He is a Senior Research Scientist at the Duke Uni-versity Center for Neuroengineering, in Durham, NC.He has held research appointments at the Institute forthe Problems of Information Transmission, Moscow(1986–1991), the International School for AdvancedStudies, Trieste, Italy (1995–1997), and the U.S. Na-

tional Institute of Mental Health (1997–2002). His research interests includeprimate neurophysiology and brain–machine interfaces.

Zheng Li received the B.S. degree in computer sci-ence and mathematics from Purdue University, WestLafayette, IN, in 2004, and the Ph.D. degree in com-puter science from Duke University, Durham, NC, in2010.

He is a Postdoctoral Associate at the Duke Uni-versity Center for Neuroengineering, Durham, NorthCarolina. His research interests are in the computa-tional aspects of brain–machine interfaces.


Miguel A. L. Nicolelis received the M.D. and Ph.D.degrees from the University of Sao Paulo, Sao Paulo,Brazil, in 1984 and 1988, respectively.

He is the Anne W. Deane Professor of Neu-roscience with the departments of Neurobiology,Biomedical Engineering and Psychology at DukeUniversity, Durham, NC. He is the Co-Director ofDuke’s Center for Neuroengineering. He is alsoFounder and President of the Edmond and Lily SafraInternational Institute for Neuroscience of Natal,Brazil and a Fellow of the Brain and Mind Institute

at the École Polytechnique Fédérale de Lausanne, Switzerland. He has authored

over 170 manuscripts, edited numerous books and special journal issues, andholds three U.S. patents.

Dr. Nicolelis’ research was highlighted in MIT Review’s Top EmergingTechnologies, and he was named one of Scientific American’s Top 50 Tech-nology Leaders in America. Other honors include the Whitehead ScholarAward; Whitehall Foundation Award; McDonnell-Pew Foundation Award; theRamon y Cajal Chair at the University of Mexico and the Santiago GrisoliaChair at Catedra Santiago Grisolia. He was awarded the International BlaisePascal Research Chair from the Fondation de l’Ecole Normale Supérieure andthe 2009 Fondation IPSEN Neuronal Plasticity Prize. He is a member of theFrench and Brazilian Academies of Science.

virtual active touch using randomly patterned intracortical microstimulation

Documents