Thesis for the Degree of Licentiate of Engineering

Embedded Controller for Arti�cial Limbs

Enzo Mastinu

Department of Electrical EngineeringDivision of Signal Processing and Biomedical Engineering

Chalmers University of Technology

Göteborg, Sweden 2017

Embedded Controller for Arti�cial Limbs

Enzo Mastinu

c© Enzo Mastinu, 2017.

Technical report number: R008/2017ISSN 1403-266X

Department of Electrical EngineeringDivision of Signal Processing and Biomedical EngineeringChalmers University of Technology

SE�412 96 GöteborgSwedenTelephone: +46 (0)31 � 772 1000Email: enzo@chalmers.se

Front Cover: The �gure on the front cover is composed by an illustration of theArti�cial Limb Controller (ALC) connected to a prosthetic arm and to the Osseoin-tegrated Human-Machine Gateway (OHMG). It also includes a picture of the �rstOHMG patient grasping a delicate item while blinded.ALC illustration and front cover graphics by Jason Millenaar and Sara Manca.

Typeset by the author using LATEX.

Chalmers ReproserviceGöteborg, Sweden 2017

Dedicated to the memories of my mother Rosanna(1960-2016) and my friend Luca (1986-2016).

Abstract

Promising developments are currently ongoing worldwide in the �eld of neuropros-thetics and arti�cial limb control. It is now possible to chronically connect a roboticlimb to bone, nerves and muscles of a human being, and use the signals sourced fromthese connections to enable movements in the arti�cial limb. It is also possible to sur-gically redirect a nerve, deprived from its original target muscle due to amputation,to a new target in order to restore the original motor functionality. Intelligent signalprocessing algorithms can now utilize the bioelectric signals gathered from remainingmuscles on the stump to decode the motor intention of the amputee, providing anintuitive control interface. Unfortunately for patients, clinical implementations stilllag behind the advancements of research, and the conventional solutions for amputeesremained basically unchanged since decades. More e�orts are therefore needed fromresearchers to close the gap between scienti�c publications and hospital practices.

The ultimate focus of this thesis is set on the intuitive control of a prostheticupper limb. It was developed an embedded system capable of prosthetic control viathe processing of bioelectric signals and pattern recognition algorithms. It includes aneurostimulator to provide direct neural feedback modulated by sensory informationfrom arti�cial sensors. The system was designed towards clinical implementation andits functionality was proven by amputee subjects in daily life. It also constitutes aresearch platform to monitor prosthesis usage and training, machine learning basedcontrol algorithms, and neural stimulation paradigms.

i

ii

Preface

This thesis is a ful�llment for the degree of Licentiate of Engineering at ChalmersUniversity of Technology within the context of the Author's industrial doctorateproject.

The work resulting in this thesis was carried out between October 2014 andJune 2017 between the Biomechatronics and Rehabilitation Laboratory within theBiomedical Signals and Systems Research Unit at the Department of Electrical En-gineering of Chalmers, and Integrum AB, Mölndal.

Professor Bo Håkansson is the examiner. In addition, Assistant Professor MaxOrtiz-Catalan and Associate Professor Sabine Reinfeldt are main supervisor and co-supervisor, respectively.

This work was �nancially supported by the Swedish Research Council (Veten-skapsrådet, D0574901), the Swedish Governmental Agency for Innovation Systems(VINNOVA), and the European Commission (H2020-ICT-2015, DeTOP project -GA 687905).

iii

iv

Acknowledgment

I would like to extend my sincere gratitude:

To my supervisors, Bo, Max and Sabine. Their invaluable help and constant sup-port are as wide as the trust they set on me, on my ideas and on my way of working.In them, I found extremely inspiring mentors and, at the same time, good friends. Atthe �rst interview for the doctorate position, Max could see beyond the appearanceof my shy and confusing English, he gave me the trust and the responsibilities neededfor my professional and personal growth.

To Rickard Brånemark and Integrum AB, for hosting me and my doctorateproject, allowing me to, somehow, be part of the amazing technological revolutionaround the idea of osseointegration.

To "magic" Jurek Lamkiewicz and Jason Millenaar, for being exactly what anelectronic engineer needs: someone who can secure the otherwise meaningless elec-tronic boards into a solid enclosure, and give them a chance to function outside mydesk. Your expertise and constant help is everyday very much appreciated.

To Stewe Jöhnson and Janos Kalmar, for their valuable support regarding pros-thetic components and clinical applications. Always helpful when discussing anypossible (and sometimes even impossible) solution.

To Pascal Doguet, Yohan Botquin and Jean Delbeke, for sharing their preciousand inspiring knowledge in matter of electronics, neurostimulators and nerves' stim-ulation.

To Fredrik Ekåsen, Alejandra Zepeda and Sonam Iqbal, for their precious sup-port and all the time spent together in designing, prototyping, testing and discussingsafety standards.

To all patients taking part in the OHMG study, for their endurance and positiveattitude, a constant lesson for my human and professional development.

v

To all researchers and scientists who gave, and gives everyday, their contributionto the �eld, with sincere apologizes to those who, erroneously, were left out from thecitations' list.

To Johan Ahlberg, for all the time spent together in the lab., and for his genuineand remarkable commitment to science.

To Alexander Thesle�, for sharing his knowledge about osseointegration and sup-porting in the writing process of this thesis.

To all my friends spread somewhere between Chalmers, Integrum, Italy and thewhole Europe. Thanks for sharing with me amazing nights, trips, music, beers, bandpractices, your happiness or sadness and long discussions about right or wrong deci-sions, beach volley and mini golf sessions, BBQs, soccer games, etc, and for constantlysurvive to my stupid jokes. You are my second family.

Alla mia famiglia, per ogni cosa. A mio padre, mio fratello e mia sorella, per es-sere ancora in piedi nonostante tutto. A mia madre, per avermi sempre incoraggiatoa seguire la mia strada, proteggendomi e spingendomi via dai problemi reali. Latua positivita' o il tuo sorriso forzato su Skype per mascherare i dolori della tua con-dizione, resteranno lezioni di vita che portero' sempre con me. A te dedico questa tesi.

Lastly, to Veronica. For being her and there, as always, still following and sup-porting my dreams, as always, and for coming in a country far away to rescue mymental health from a certain breakdown. Grazie Veno'.

vi

List of Publications

This thesis is based on the work contained in the following appended papers:

Paper I

Mastinu, E., Ortiz-Catalan, M., and Håkansson, B., �Analog Front-Ends comparisonin the way of a portable, low-power and low-cost EMG controller based on PatternRecognition�, Proceedings of the 37th Annual International Conference of the IEEEEngineering in Medicine and Biology Society, August 25-29, 2015, Milano, Italy, vol.64, no. 4, pp. 2263�2269, 2016.

Paper II

Mastinu, E., Håkansson, B., and Ortiz-Catalan, M., �Low-cost, open source bioelec-tric signal acquisition system�, Proceedings of the 14th Annual International Confer-ence on Wearable and Implantable Body Sensor Networks of the IEEE Engineeringin Medicine and Biology Society, May 9-12, 2017, Eindhoven, Netherlands, vol. 26,no. 4, pp. 261-263, 2017.

Paper III

Mastinu, E., Doguet, P., Botquin, Y., Håkansson, B., and Ortiz-Catalan, M., �Em-bedded System for Prosthetic Control Using Implanted Neuromuscular InterfacesAccessed Via an Osseointegrated Implant�, IEEE Transactions in Biomedical Cir-cuits and Systems, vol. 11, no. 4, pp. 867-877, 2017.

Paper IV

Mastinu, E., Ahlberg, J., Lendaro, E., Håkansson, B., and Ortiz-Catalan, M., �Anovel approach to myoelectric pattern recognition for the control of hand prostheses:A case study of use in daily life by a dysmelia subject�, Submitted to IEEE Journalof Translational Engineering in Health and Medicine, 2017.

vii

Other related publications of the Author not included in this thesis:

• Mastinu, E., Ortiz-Catalan, M., and Håkansson, B., �Digital Controller for Ar-ti�cial Limbs fed by Implanted Neuromuscular Interfaces via Osseointegration�,Proceedings of the 38th Annual International Conference of the IEEE Engineer-ing in Medicine and Biology Society, August 16-20, 2016, Orlando, Florida,U.S.A., 2016.

• Mastinu, E., Håkansson, B., and Ortiz-Catalan, M., �Digital Controller for Ar-ti�cial Limbs fed by Implanted Neuromuscular Interfaces via Osseointegration�,Proceedings of the Trent International Prosthetic Symposium, September 28-30,2016, Glasgow, Scotland, U.K., 2016.

• Mastinu, E., Håkansson, B., and Ortiz-Catalan, M., �Embedded Controller forPattern Recognition and Neural Stimulation via Osseointegration�, Proceedingsof the 16th World Congress from the International Society for Prosthetics andOrthotics, May 8-11, 2017, Cape Town, South Africa, 2017.

• Lendaro, E., Mastinu, E., Håkansson, B., and Ortiz-Catalan, M., �Real-timeclassi�cation of non-weight bearing lower limb movements using EMG to fa-cilitate phantom motor execution: engineering and case study application onphantom limb pain�, Accepted for publication on Journal of Frontiers in Neu-rology, section Neuroprosthetics, 2017.

• Gusman, J., Mastinu, E., and Ortiz-Catalan, M., �Evaluation of Computer-Based Target Achievement Tests for Myoelectric Control� Submitted to IEEEJournal of Translational Engineering in Health and Medicine, 2017.

• Naber, A., Mastinu, E., and Ortiz-Catalan, M., �Stationary Wavelet Processingand Data Imputing in Myoelectric Pattern Recognition on a Low-Cost Embed-ded System�, Manuscript ready for submission to Journal of NeuroEngineeringand Rehabilitation, 2017.

viii

Abbreviations and Acronyms

ALC Arti�cial Limb Controller

DC Direct Control

DoF Degrees-of-Freedom

DSP Digital Signal Processor

EMG Electromyography

ENG Electroneurography

IMES Implantable Myoelectric Sensor

MCU Microcontroller Unit

MPR Myoelectric Pattern Recognition

OHMG Osseointegrated Human-Machine Gateway

OPRA Osseointegration Prostheses for the Rehabilitation of Amputees

FPGA Field Programmable Gate Arrays

RPNI Regenerative Peripheral Nerve Interface

TMR Targeted Muscle Reinnervation

TSR Targeted Sensory Reinnervation

ix

x

Contents

Abstract i

Preface iii

Acknowledgments v

List of Publications vii

Acronyms ix

Contents xi

I Introductory Chapters

1 Introduction 1

1.1 Scope and Structure of the Thesis . . . . . . . . . . . . . . . . . . . . 1

2 Conventional Clinical Solutions 3

2.1 Suspended Sockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Body-Powered Prostheses . . . . . . . . . . . . . . . . . . . . . . . . 42.3 Myoelectric Prostheses . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Advanced Surgical Techniques 7

3.1 Targeted Muscle Reinnervation and Targeted Sensory Reinnervation . 73.2 Regenerative Peripheral Nerve Interface . . . . . . . . . . . . . . . . . 93.3 Transplantation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.4 Osseointegration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4 Control Strategies for Myoelectric Prostheses 13

4.1 Direct Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

xi

Contents

4.2 Myoelectric Pattern Recognition . . . . . . . . . . . . . . . . . . . . . 14

5 Upper Limb Myoelectric Pattern Recognition Systems 17

6 Neural Interfaces for Myoelectric Prostheses 19

6.1 Percutaneous Leads . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206.2 Wireless Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.3 Osseointegrated Human-Machine Gateway . . . . . . . . . . . . . . . 22

7 Sensory Feedback and Closed-Loop Control 23

8 Summary of the Thesis Contributions 25

9 General Conclusions and Future Work 27

II Included Papers

xii

Part I

Introductory Chapters

Chapter 1Introduction

The human hand consists of 27 bones, 28 muscles, 3 major nerves, multiple tendons,as well as arteries, veins and soft tissue. It is an incredibly complex system witha huge spectrum of functionality. Hands are essential not only to interact withdi�erent objects daily, but also necessary for social interactions, such communicationand arts. The loss of a hand is a terri�c traumatic experience, usually followedby signi�cant psychological and rehabilitation challenges. The interaction betweenengineering and science has, since a long time, been pointed towards the restaurationof the functionality of a lost limb, and this thesis aims to contribute to such goal.

1.1 Scope and Structure of the Thesis

This Licentiate thesis is focused on the natural and intuitive control of an arti�ciallimb to replace the lost functionality in cases of upper limb amputation. Electromyo-graphic (EMG) (also de�ned as myoelectric) signals and their application for pros-thetic control is an imperative background concept for this thesis.

Most of the e�orts gathered in this thesis is a logic consequent step to a previousstudy where an Osseointegrated-Human-Machine-Interface (OHMG) was developedand implanted on a pilot subject [1]. The OHMG achieved a long-term interface tobone, muscles and nerves of a patient thanks to an osseointegrated titanium implant,epimysial and cu� electrodes, and bidirectional feedthrough mechanisms. Osseointe-gration creates a stable mechanical attachment of the prosthesis, while the implantedelectrodes provide long-term stable access to bioelectric signal sources and sites forperipheral nerves stimulation. The work carried within this thesis should be seen, inpart, as an answer to a particular demand: the need of an advanced electronic controlsystem compatible with the OHMG, and capable of state-of-the-art processing algo-rithms, and of direct neural stimulation. Thereby, the focus was on the developmentand the validation of an embedded system needed to exploit the advantages of the

1

Chapter 1. Introduction

OHMG for closed-loop prosthetic control, meaning to concurrently enable intuitivemotor control and sensory feedback. Moreover, it was shown that it can be used incombination with implanted electrodes (within the context of the OHMG) as well aswith less invasive sensors (surface EMG electrodes). Lastly, in order to promote theresearch in the �eld, an open-source complete package (hardware and software) foradvanced investigations into myoelectric prosthetic control was made available in thescienti�c community.

The �rst part of the thesis is structured as an introduction to the �eld providingthe reader of some of the background knowledge needed for the attached articles.Chap. 2 presents the current conventional clinical solutions for prosthetic applica-tions while Chap. 3 introduces the reader to some of the most advanced surgicaltechniques available today for amputees. Chap. 4 brie�y points out the contrapo-sition between the conventional control strategy for operating arti�cial upper limbsand the state-of-the-art strategy proposed lately by researchers (myoelectric patternrecognition), de�ned as capable of providing natural prosthetic control, and of which,some applications and examples are furtherly reported on Chap. 5. Chap. 6 collectssome of the main challenges and achievements in matter of neural interfaces, whileChap. 7 describes the concept of closed-loop prosthetic control.

In the second part of this report, four scienti�c articles developed within the timeof this thesis are included. A brief description of their contributions can be found inChap. 8.

2

Chapter 2Conventional Clinical Solutions

Despite the amount of research spent over the last decades in the �eld of prosthetics,the main solutions clinically available for patients remained basically unchanged in thelast 40 years. In order to restore the functionality of a lost limb, two challenges needto be addressed: how to attach the prosthesis to the body, and how to functionallycontrol the arti�cial limb. Suspended sockets are usually provided to patients tosecure, by compression of the skin, the prosthetic extremity. The terminal devicescan then be driven via either a system of cables (body-powered) or via bioelectricsignal measured on the skin surface at the stump level (myoelectric), as shown fromFig. 2.2.

2.1 Suspended Sockets

The conventional method to attach prosthesis to the patient's stump is via a sus-pended socket. Sockets rely on mechanically compressing the tissue in the stump tosecure the arti�cial limb. Therefore, skin contact and friction are essential elementsfor the attachment. The socket must be custom made according to the stump of eachpatient. There are several drawbacks related to the use of suspended sockets, majorones are listed in the following:

• skin irritation or in�ammation

• poor �t and mechanical reliability

• limited range of motion

• limitations in use due to environment temperature conditions

• sweating causing unpleasant smelling.

3

Chapter 2. Conventional Clinical Solutions

Depending on the patient and his/her level of physical activity, these problems canescalate from an uncomfortable situation to a point where they can actually preventthe patient to wear the prosthesis. Today, suspended sockets are widely identi�ed asone of the major source of issues for amputees all over the world.

2.2 Body-Powered Prostheses

The concept of an upper limb prosthesis driven by remaining parts of the body waspioneered in Germany at the beginning of the 19th century. This device representsthe �rst documented example of a so called �body-powered� prosthesis. It relied ontrunk movements transferred from a shoulder girdle to a terminal device throughleather straps (Fig. 2.2, left). Since then, the biggest technologic improvement forbody-powered prostheses came in 1948, when the �rst device using a Bowden cablewas introduced. The body-powered prostheses available today are essentially opti-mizations of that design [2].

Regardless the simplicity of their design and the not-anthropomorphic appear-ance (most commonly a two-pronged hook), body-powered prostheses are still widelydi�used in the amputees' community [3]. This longevous success is due to their highvalue for money. They are lightweight, robust and require relatively simple mainte-nance; a skilled user can achieve an impressive range of functional motions, moreover,to some extent force discrimination and proprioperception is inherently possible bysensing the cable tension.

In 2016, the �rst edition of the Cybathlon was held in Switzerland (Fig. 2.1),meant as an Olympic-inspired competition for people with disabilities exploiting as-sistive technologies. Interestingly, the upper limb prosthesis category was won by abody-powered user. Even though this event was mostly meant as a competition forrobotic devices, the commitment to this event of the winning team was actually toproof the still superior e�cacy of body-powered prostheses compared to state-of-the-art advanced robotic devices.

2.3 Myoelectric Prostheses

The mechanical e�ort to maneuver a body-powered prosthesis can often be fatiguing.Electrically-powered arti�cial limbs try to solve this by actuating motors throughan electronic control system (Fig. 2.2, right). These devices, commonly de�ned asmyoelectric prostheses, are based on the utilization of electrical activity measured atthe stump from the remaining muscles (myoelectric signals), which is triggered bythe user to drive the prosthesis. Myoelectric prostheses were �rstly proposed in the�rst half of the 20th century. Some early studies were carried out by De Luca etal. (starting from the 1970's) about the scienti�c de�nition of neuromuscular signals

4

2.3. Myoelectric Prostheses

Figure 2.1: Medal ceremony of Cybathlon 2016 - ARM category (image from Cy-bathlon webpage). A body-powered prosthesis user won, followed by a myoelectricuser operating one of the most advanced terminal devices, followed by the OHMGuser operating a standard myoelectric device.

5

Chapter 2. Conventional Clinical Solutions

Figure 2.2: Representation of two typical solutions for below-elbow prosthesis [5].Left) Body-powered Bowden cable prosthesis controlled by �gross� movements of thebody. Right) Myoelectric prosthesis controlled by electomyographic (EMG) signalcaptured from residual muscles at the stump.

and the challenges behind their use for prosthetic control [4]. Indeed, due to criticaltechnology limitations, myoelectric prostheses became a clinically valid solution onlyaround the 1980's [2]. Since then, the technology has remained essentially the same.

Source electrodes are commonly non-invasive and placed over the skin surfaceof two groups of antagonist muscles. Proportional control is allowed by varyingthe intensity of the muscles' activity. Besides some �incidental clues� like noise orvibrations from electric motors, sensory feedback is still not part of a standard my-oelectric device prescription and therefore, visual feedback is constantly required toproperly operate these devices. Most commonly, their aesthetics is superior of anybody-powered prosthesis; the mechanics can be hidden under life-like hand siliconegloves available in di�erent skin tonalities. The primary disadvantages of this typeof prostheses are currently their cost and weight, but fragility and maintenance arealso major concerns.

The biggest breakthrough of myoelectric terminal devices is represented by theadvent of multi-functional robotic hands. Manufacturers started recently to equiptheir robotic terminal devices with microprocessors which allow for more complexfunctioning. Hand with multiple grips or postures are now an option on the market.Each pre-de�ned position can be reached by triggering some pre-de�ned pattern ofEMG activity, e.g., a series of triple impulses on the �open hand� control signal. Thesegestures are meant to facilitate the user manipulation of items through functioning�ngers.

6

Chapter 3Advanced Surgical Techniques

Considering the relative low e�cacy of standard clinical solutions and their major lim-itations, alternative approaches have been suggested over the years from researchers,typically, involving a surgical procedure. The outcomes of a modern hand transplantssurely identify it as one of the most promising and intriguing techniques. Alterna-tively, amputated nerves can be now redirected to new target muscles for both controland sensorial purposes. Osseointegration allows for direct skeletal anchorage of theprosthesis, its popularity is growing fast and it is often recognized as the future inthe prosthetic �eld.

3.1 Targeted Muscle Reinnervation and Targeted

Sensory Reinnervation

There are precise technology challenges regarding the recording of activity from theperipheral nervous system. Neural action potentials have characteristic amplitude offew micro volts and their direct use for prosthetic control imposes hard technologicalrequirements. Due to amputation, some nerves are deprived of their original targetmuscle. In 1980, Ho�er and Loeb suggested a novel surgical technique to naturallyamplify neural signals via reinnervating a cut nerve on a new target muscle, namelyTargeted Muscle Reinnevation (TMR). After the reinnervation process settles, it ispossible to use these new sites for prosthetic control via EMG acquisition from thesurface of the skin. This idea, represented in Fig. 3.1, had its �rst clinical imple-mentation in 2004 thanks to Kuiken et al., where TMR allowed a bilateral shoulderarticulation patient the control of a 3 Degrees-of-Freedom (DoF) prosthesis [6]. TheTMR technique was more formally assessed by Kuiken et al. in 2009, where residualarm nerves where successfully transferred to alternative muscle sites in �ve patientsand their ability in controlling a virtual prosthetic limb was measured, together withexperimentations over multiple-DoF prostheses [7].

7

Chapter 3. Advanced Surgical Techniques

Figure 3.1: Representation of Targeted Muscle Reinnervation from Kuiken et al. [7].

It has been estimated that over 60 patients have been treated with this proceduresince 2002.

An unexpected outcome of the �rst TMR patients was a sensory recovery on thepreceptors of the skin overlaying the reinnervation sites. This was further investigatedby extending the TMR to reinnervate sensory nerves to the main peripheral nervetrunk, developing a new surgical procedure de�ned as Fascicular Targeted SensoryReinnervation (TSR). The idea is to create a discrete spatial sensory hand map ona skin area relatively distant from the prosthesis. Early results on a single patientshowed the e�ectiveness of the TSR to recover pressure sensation discrimination onamputees [8].

8

3.2. Regenerative Peripheral Nerve Interface

Figure 3.2: Representation of Regenerative Peripheral Nerve Interface (image from[10]). Small portions of muscles can be transplanted to serve as a target for nervesdeprived of a functional target muscle after amputation.

3.2 Regenerative Peripheral Nerve Interface

The TMR technique has been further improved in the so called Regenerative Periph-eral Nerve Interface (RPNI) [9, 10]. Here, small portions of muscles can be trans-planted to serve as a target for nerves deprived of a functional target muscle afteramputation (Fig. 3.2). Multiple transplants can be executed on the same patient tocreate a matrix that can be further used for EMG acquisition and prosthetic control.Due to the small portion of muscle transplanted, this technique imposes the use ofimplanted electrodes for EMG acquisition for a reliable control of the prosthesis [11].The RPNI has been tested successfully on both animal models and humans as atreatment for post-amputation neuroma pain.

9

Chapter 3. Advanced Surgical Techniques

Figure 3.3: Representation of a hand transplantation (image from John HopkinsMedicine University, Comprehensive Transplant Center webpage).

3.3 Transplantation

Hand and upper limb transplantation represents certainly one of the most fascinatingways to restore the cosmesis and functionality of the lost peripheral limb. It is acomplex surgical procedure which aims to transfer a hand (or a full forearm) froma donor to a recipient. The surgery can last from 8 to 12 hours and it involvesbone �xation, reattachment of arteries, veins, tendons, nerves and skin (Fig. 3.3). Itis followed by a heavy immunosuppression medication and a tedious rehabilitationprocedure, which results can strongly vary from a patient to another. That is whypatient selection is widely considered the most important aspect of the transplanttechnique, where a special emphasis on medical, behavioral, psychological, socialfactors, as adherence to immunologic and rehabilitative therapy is mandatory toachieve optimal outcomes [12].

Several failures are associated with the �rst reported attempts of this surgery,mostly due to primitive and insu�cient immunosuppression medications and rehabil-itation post-transplantation. This technique is not considered experimental anymore:its improvement in the quality of life is heavily recognized thanks to the modern out-comes in the matter of functionality and survival time (longest-lasting period of 16years, [12]).

10

3.4. Osseointegration

3.4 Osseointegration

Bone-anchored prostheses are a solution to the drawbacks related to coupling arti�ciallimbs via suspended sockets (described in Section 2.1) which is the conventional wayof attaching limb prostheses. Bone-anchored prostheses allow for direct transfer ofexternal loads from the arti�cial limb to the skeleton, eliminating the need of asuspended socket. Direct skeletal attachment of the prosthesis is currently pursuedthanks to the concept of osseointegration. Osseointegration was de�ned as �closeadherence of living bone tissue to an implant surface without intervening soft tissue,thus allowing for a structural functional connection between a load bearing implantand the bone tissue� [13]. A strong structural connection between living bone tissueand a foreign material is thus possible, provided the biocompatibility of that material.It was discovered in the early 1960's by Dr. P. I. Brånemark in Gothenburg, Sweden,and now it is a world-wide consolidated clinical practice for dental implants. Itsapplication for arti�cial limb attachment started in the 1990's with titanium custommade implants that consequently led to the establishment of the OsseointegrationProstheses for the Rehabilitation of Amputees (OPRA) Implant System (IntegrumAB, Mölndal, Sweden). The OPRA (Fig. 3.4) is based on the implantation of atitanium externally-threaded cylindrical platform (�xture) into bone tissue in thestump. Another titanium unit (abutment) is then �xated (press-�t) into the �xtureextending percutaneously from the residual limb and allowing for attachment of anexternal prosthesis. A third element (abutment screw) is then used to clamp andsecure the abutment to the �xture. The OPRA Implant System was initially meantfor lower limbs amputation, but it was shortly after applied to upper limb as well.

Other typologies of implants and anchoring technologies have been proposedaround the world and are currently available, or under development, or under clinicalvalidation. Some examples are:

• Integral Leg Prosthesis (ILP), Orthodynamics GMbH, Lübeck, Germany

• Osseointegrated Prosthetic Limb (OPL), Permedica s.p.a., Milan, Italy

• Intraosseous Transcutaneous Amputation Prosthesis, Stanmore Implants World-wide (ITAP), Watford, United Kingdom

• Keep Walking Advanced, Tequir S.L., Valencia, Spain

• Percutaneous Osseointegrated Prosthesis (POP), DJO Global, Austin, USA

• COMPRESS, Zimmer Biomet, Warzaw, USA

A wide collection of scienti�c publications (mostly regarding the �rst Swedishsystem) is validating the improvement of the quality of life for bone-anchored am-putees [14, 15]. Additional bene�ts of direct skeletal attachment of prosthetic limbsinclude:

11

Chapter 3. Advanced Surgical Techniques

Figure 3.4: The OPRA Implant System. Left) Schematic of the implant (image fromIntegrum AB webpage). The OPRA implies the implantation of a titanium platform(�xture) into bone tissue in the stump. Another titanium unit (abutment) is then�xated into the �xture and extends percutaneously, while a third element (abutmentscrew) is then used to clamp the abutment to the �xture. Right) Clinical photographshowing a patient with the osseointegrated implant attached to an external prosthesis[16].

• improved range of motion (for both upper and lower limbs)

• improved walking ability and reduced energy expenditure

• improved comfort during sitting position

• improved awareness via osseoperception.

Osseointegration, like any other surgical procedure, implies risks. Super�cial in-fections can arise at the percutaneous interface, requiring an antibiotics treatment.Deeper infections are rare but, if not promptly treated, can led to the implant re-moval [16,17].

The commercial interest in this technology is growing fast as well as the list ofcountries where the treatment is available. Moreover, the FDA of U.S.A. recentlyapproved the �rst bone-anchored implant (OPRA Implant System) and surgical pro-cedure, and the U.S. Army has started a clinical trial, where lower limb amputeesare treated with this system.

12

Chapter 4Control Strategies for Myoelectric

Prostheses

Despite the advances in prosthetic hardware that allow an increasing number ofarti�cial joints to approach those of the lost limb (Modular Prosthetic Limb, [18],DEKA arm, [19]), a major issue remains unsolved, namely, how to achieve a reliableand natural control of the prosthetic limb. After decades of research and developmenton upper limb myoelectric prosthetics, current clinical applications for amputees stillrely on a 30-years-old strategy for control, namely direct control (DC) [20].

4.1 Direct Control

Direct Control, also known as one-for-one or one-muscle-to-one-function, implies theutilization of recti�ed myoelectric signal from two groups of antagonist muscles (e.g.,biceps and triceps for above elbow amputation, �exors and extensors for below el-bow) to trigger the terminal prosthetic device (Fig. 4.1). For instance, hand openand close can then be driven via a set of two complementary movements (e.g., elbow�ex and extend for above elbow amputation, wrist �ex and extend for below elbow).Proportional control, de�ned as speed-controllable movements, is then achieved bymodulating the intensity of the electrical activity measured at the stump: a strongcontraction will be interpreted and converted as a fast prosthetic activation of thatparticular movement. For cases in which limited control sites are available, a con�gu-ration of one-muscle-to-two-functions is also possible, by matching di�erent activationthresholds to di�erent prosthetic movements (e.g., a weak and a strong elbow �exionfor hand open and close). Adaptation is usually required to get the user familiarwith the new set of muscles-contraction and expected-limb-movement. The DC ap-proach is pervasive mostly owing to its simplicity, relative good reliability, and easeto learn. Unfortunately, the functional outcome is commonly related to the speci�cpatient predisposition, thus often resulting in rejection of the myoelectric prosthesis,

13

Chapter 4. Control Strategies for Myoelectric Prostheses

Figure 4.1: Representation of Direct Control strategy for myoelectric prostheses. Itimplies the utilization of recti�ed myoelectric signal from two groups of antagonistmuscles (e.g., �exors and extensors for below elbow amputation) to trigger the ter-minal prosthetic device.

or in reduction of the robotic potential (independent �ngers control, wrist rotation,elbow joint angulation) to a simple prosthetic claw [21]. Di�erent variations of thestandard DC approach are available nowadays to provide more controllable DoF, butunfortunately, always with a cost of improved complexity and reduced intuitivenessof the control interface.

4.2 Myoelectric Pattern Recognition

Around the mid-1960's the �eld of neural prosthetics started to involve computerintelligence for improving the control experience for the user (and to reduce thetraining burden) [22]. The need of a more natural control interface met the growingpotential of pattern recognition algorithms. Pattern recognition (namely also machinelearning) is an umbrella term which covers a variety of algorithms, ranging fromstatistical to biologically inspired, which have in common the same task: identifypatterns or regularities in data and consequently recognize them when a new sampleof data is presented. These algorithms are usually de�ned as supervised, when thedecision regarding new data is made in reference to analogous pre-labeled data usedas a training set, or unsupervised, when no pre-labeled data is available. Moreover,we di�erentiate between classi�cation and regression problems, when the decision ofthe algorithm has the form of a class (or a label), or continuous values, respectively.

14

4.2. Myoelectric Pattern Recognition

Figure 4.2: Representation of myoelectric pattern recognition strategy for a classi�-cation problem. The acquired EMG signals are windowed and a proper set of featuresis then extracted to reduce the dimensionality of the problem. These features are feedthe classi�er, which was previously trained with analogous data (supervised learning),to classify the motor intention of the user.

Natural control is de�ned in this context as the ability of providing control ofthe prosthesis in the same way as an intact physiological system would do, thusintuitive and spontaneous movements of the phantom arm are properly translated tothe arti�cial limb. This approach is often de�ned as myoelectric pattern recognition(MPR), meaning machine learning applied for motor volition prediction using theEMG as input signal. Single-muscle targeting is not necessary for MPR, but insteadmultiple channels are spread over the stump to gather as much useful informationneeded to characterize (and thus di�erentiate) each movement.

The �rst investigations of MPR started in the mid-1960's at the RehabilitationEngineering Center of Philadelphia, and a complete report was published later byWirta et al. [22] (Fig. 5.1). A multivariate statistical program was used to classify fourmovements fed with myoelectric signal previously recorded from six surface EMG sitesfrom an able-bodied subject. Shortly after in 1973, Herberts et al., reported MPRapplied for the simultaneous control of a three DoF prosthesis [23]. The DiscriminantAnalysis in linear con�guration (LDA) was deemed as the classi�cation algorithm andcurrently, it is still one of the most renowned tools for MPR researchers. It was also inthose same years (mid-1970's), that MPR researchers started to realize that targetingthe muscles was not necessarily the optimal electrodes con�guration anymore, butinstead it was more convenient to �nd other con�gurations able to exploit the mostof the information available on the stump (e.g., electrodes in between muscles [24]).

15

Chapter 4. Control Strategies for Myoelectric Prostheses

MPR became slowly more and more popular among the prosthetic research �eldbecause of its appealing potential [25,26]. Generally, all major promising algorithmsdeveloped over the years in the machine learning �eld were exploited by researchersfor prosthetic MPR applications. Some examples are:

• Neural Network,

• Support Vector Machine,

• K-Nearest Neighbor,

• Gaussian Mixture Model.

More recently, modern deep learning algorithms started to be used and evaluated forprosthetic control purposes [27,28].

Even though MPR represents today the state-of-the-art for prosthetic controlinterface, its clinical implementation is still far from being a concrete reality for am-putees. Up to date, there is only one commercially available MPR system, namelyComplete Control from COAPT (Chicago, Illinois, U.S.A.), and although clinicalinvestigations are ongoing, no results have been made publicly available in the sci-enti�c community. A still high rejection rate appears to be towards this technology,from both patients and clinicians. Potential reasons for the slow take o� of MPRare due to well-known di�culties related to multiple channels surface EMG acquisi-tion. Issues such as motion artifacts, electromagnetic interferences, frequent changesat the interface skin-electrode, socket manufacturing cost, contribute to make MPRstill unpractical for wide dissemination.

16

Chapter 5Upper Limb Myoelectric Pattern

Recognition Systems

The �rst MPR embedded system for the control of a prosthetic arm was developed inthe mid-1960's and it was a simple weighted network of resistors (Fig. 5.1) [22]. Forthe �rst time, a MPR system was able to recognize di�erent movements taking asinput EMG activity recorded simultaneously from di�erent channels. More precisely,it was fed by six EMG sites and capable of predicting four movements with reportedaccuracy around 90%. Shortly after, Herberts et al., reported a similar pure-analogicMPR system applied for the simultaneous control of a three DoF prosthesis [23].The battery-powered portable system was able to drive the robotic arm for handprehension, wrist rotation and elbow �exion/extension and tested on an above-elbowamputee.

Opposed to portable systems, computer-based MPR systems started to appearin the literature around the 1980 [29]. Computer-based platforms were, and stillremains, a very proli�c source for literature studies, used as a test-bench for MPRalgorithms and control performance [7,30,31], but also for �serious gaming�, trainingand rehabilitation purposes [32], and for phantom limb pain reduction [33].

The advent of microprocessors and of microcontroller units (MCU) was a crucialbreakthrough towards any modern portable and wearable device, and this includesprosthetic controller as well. The �rst attempt of a MCU-based MPR system is dated1977, included in a work published from Graupe et al. [24]. It relied on one of the�rst MCU, the 8080 from Intel (Santa Clara, California, U.S.A.), for real time autore-gressive analysis of EMG signal to classify motions and to properly actuate a roboticdevice. The reported classi�cation accuracy was between 85% and 95%. The intereston MPR prosthetic control systems grew constantly and developing in parallel withthe computation capabilities of new emerging processing units. An interesting dis-cussion was carried out by Xiao et al. about the advantages of having a GPU corefor MPR prosthetic applications [34]. Approaches using Field-Programmable Gate

17

Chapter 5. Upper Limb Myoelectric Pattern Recognition Systems

Figure 5.1: The �rst embedded system employing pattern recognition for the controlof a prosthetic arm [22]. Developed in the mid-1960's, it was composed by a simpleweighted network of resistors.

Arrays (FPGA) have shown to be highly bene�cial for accelerating the computa-tion of pattern recognition algorithms [35]. FPGAs represent a valuable solution forprosthetic control which is predestined to appear in future embedded MPR systems,but currently no clinical implementation of such system has been reported. Digi-tal signal processor (DSP), like the popular series TMS320 from Texas Instruments(Dallas, Texas, U.S.A.), have been selected for the development of portable MPRsystems [36,37]. A portable speech recognition DSP-based system was also suggestedby Lin et al., as a solution for prosthetic control [38]. Other system designs involvedmore powerful but also more power demanding processors like the PXA270 from In-tel or the CortexA8 from ARM (Cambridge, England, U.K.) [39,40]. This approachobviously poses energy consumption challenges but still provides a remarkable pro-cessing capability for a wearable device. Processor cores from the Cortex-M familyrecently started to be a popular choice for MPR systems' design given their e�cientcompromise between power consumption and computation capabilities [41]. This corewas also chosen for the system presented in Paper III attached in this thesis. To datethere is a single commercially available embedded MPR system from which limitedinformation is available due to its commercial nature (Complete Control, COAPT).

18

Chapter 6Neural Interfaces for Myoelectric

Prostheses

The functional challenges of any myoelectric prosthesis, MPR or not, are related tothe sensors used to acquire the EMG signals. Clinical solutions typically includenon-invasive electrodes, where conductive parts are located on the surface of the skinabove the target muscles. The selectivity of this approach is limited due to the signaltraveling through di�erent tissues before reaching the sensing parts or cross-talkbetween neighboring muscles. Any electrode misplacement or electromagnetic �eldin proximity can potentially create artifacts able to involuntary drive the prosthesis.Moreover, environmental conditions (humidity and temperature) cause impedancechanges in the interface skin-electrode which can contribute to reduce the reliabilityof these systems and force to exhaustive recalibration.

Since all these aspects can greatly threat the overall functionality of non-invasivemyoelectric devices, several invasive approaches have been investigated to potentiallysolve these problems. Invasive solutions basically rely on implanted electrodes forsensing the myoelectric signal. Several con�gurations for implanted neuromuscularelectrodes are available today. Besides all the risks involved in the surgery and inforeign-body reactions, the major system-design challenge for invasive solutions hasbeen historically how to functionally access the control signals sourced by implantedsubcutaneous parts. Several solutions have been thus proposed.

19

Chapter 6. Neural Interfaces for Myoelectric Prostheses

Figure 6.1: First attempt of a percutaneous neuromuscular interface for prostheticcontrol, 1980 [42]. A transradial amputee was implanted with four epimysial elec-trodes and a cu� electrode which leads converged on a percutaneous connector emerg-ing from the skin of the upper arm.

6.1 Percutaneous Leads

Percutaneous leads have been explored and utilized in literature, and despite not be-ing regarded as a long-term stable solution due to obvious problems at the interfacebetween leads and skin, they played a crucial role in research for neural prosthetics.The �rst pioneering attempt of a percutaneous neuromuscular interface for prostheticcontrol is dated back to 1980, reported by Ho�er and Loeb [42] (Fig. 6.1). A tran-sradial amputee was implanted with four epimysial electrodes on the muscles and acu� electrode planted around one large fascicle of the ulnar nerve (proximal to theneuroma). Leadout cables converged onto a 12-pin socket percutaneous connectoremerging from the skin of the upper arm. The di�culty in securing the cables atthe interface and prevent skin in�ammation eventually led to an infection followedby the percutaneous connector removal.

To our knowledge, the longest reported time of implantation for percutaneousimplants is over four years [43], but it is widely recognized that these solutions willeventually reach a functional failure. Currently, they do not represent a clinical viablesolution for amputees but still they are an important source for basic (and not only)science research in the �eld of neural prostheses.

20

6.2. Wireless Interfaces

Figure 6.2: Representation of the IMES system [47]. The EMG sensors are implantedin the forearm and communicate wirelessly to the external coil laminated in theprosthetic socket.

6.2 Wireless Interfaces

Provided the limitations of percutaneous solutions, wireless neural interfaces havebeen suggested and developed over the years. Implanted telemetric systems can beused to bidirectionally transfer information between the body and the robotic con-trol system. The �rst experimentation of a wireless system for prosthetic control wasdone by Herberts et al. in the 1968 [44]. Here, the six implanted electrode-capsuleswere wirelessly energized from a single external processing unit placed on the skin.EMG signals were successfully transferred via frequency modulation of the carrierwave. Two able-bodied subjects and two below-elbow amputees were recruited forexperimentation; the implantation lasted from 3 up to 15 months, and some primi-tive foreign-body reactions were reported. Few years later, in 1974, Clippinger et al.,presented a wireless neural stimulation system aimed to integrate sensory feedbackon a body-powered prosthetic system [45]. Here, oppositely to Herberts' system, allimplanted electrodes converged to a centralized transmission unit connected wire-lessly to the main external unit. The implant was removed several weeks after.The next clinical implementation of a wireless prosthetic system was not reporteduntil 2014, where Pasquina et al. provided two transradial amputee subjects withthe Implantable Myoelectric Sensor (IMES) technology [46]. The IMESs are small,cylindrical electrodes (16 mm long and 2.5 mm of diameter) capable of detecting andwirelessly transmitting EMG data [47] (Fig. 6.2). Thanks to these telemetric sensorsthe test subjects could functionally control a 3 DoF prosthetic arm.

21

Chapter 6. Neural Interfaces for Myoelectric Prostheses

Figure 6.3: Osseointegrated Human-Machine Gateway (OHMG). Left) Representa-tion of the modular system. Right) Placement of epimysial and cu� electrodes in theright upper arm [1].

6.3 Osseointegrated Human-Machine Gateway

The technology for the direct skeletal attachment of a limb prosthesis (started in1990 [17]) provided the framework for an alternative method to gain access to the pe-ripheral nervous system. In fact, osseointegration inherently predisposes of a percuta-neous interface between the skin and the titanium abutment. In 2014, Ortiz-Catalanet al. demonstrated the possibility of a long-term bidirectional communication be-tween the arti�cial limb and implanted neuromuscular interfaces by incorporatingsignal feedthrough mechanism into the osseointegrated implant [1]. Here, some of theparts constituting the OPRA Implant System were modi�ed to integrate feedthroughconnectors, which interface to the implanted electrodes' leads outside the bone in thesoft tissue. The system included epimysial electrodes, in both monopolar and bipolarcon�guration, targeting biceps and triceps muscles, as well as a cu� electrode locatedaround the ulnar nerve. This technology, named as the Osseointegrated Human-Machine Gateway (OHMG) and shown in Fig. 6.3, was implanted on a transhumeralamputee subject and it is still currently functional at the 4th year follow up. Theadvantages of implanted EMG sensors for prosthetic limb control were proven as wellas the functionality of the cu� electrode for direct neural stimulation.

Three more transhumeral amputees have recently been implanted as part of anongoing clinical trial.

22

Chapter 7Sensory Feedback and Closed-Loop

Control

Even though active prosthesis can provide an acceptable restoration of functionality,sensory feedback is still missing and not purposely pursued in clinical practice. Sev-eral surveys reported this to be a cause for preventing a wider acceptance of prostheticdevices over the amputees' community [48,49]. Exteroception (sense of the surround-ing environment) and proprioception (sense of the proper state, joint angles, etc) aremandatory requirements for any prosthetic device which wishes to be referred to as�able to provide a functional natural sensation�. Tactile feedback is essential in theinteraction with items, and researchers agree on the necessity of investigating viablemethods to provide a closed-loop control of the prosthesis. In the last decades, sev-eral approaches have been proposed but none has been yet unanimously approvedfrom clinicians to be usable and suitable for daily operation. Sensory substitutionis an approach that has been widely explored in research. It is based on the ideaof providing sensory information to the body through a sensory channel that di�ersfrom the natural one, e.g., substitute touch with hearing, or via the same channelbut in a di�erent modality, e.g., substitute pressure with vibration. Vibrotactile andelectrotactile are two examples of sensory substitution techniques investigated sincedecades [50].

Vibrotactile sensory substitution transfers information to the user by vibratingthe skin at frequencies not higher than 500 Hz [52]. The information can be mod-ulated acting on two main parameters, amplitude and frequency of the vibration.The discrimination capabilities of the user are strictly related to the location wherethe vibrating parts are applied. Electrotactile stimulation uses a local current toevoke sensations and convey information to the user [52]. Major parameters are cur-rent amplitude, shape, frequency and duration of the pulse waveform. Oppositelyto vibrotactile, no mechanical parts are involved, therefore, electrotactile-capable de-vices are characterized by low power consumption and fast response. Mechanotactile

23

Chapter 7. Sensory Feedback and Closed-Loop Control

Figure 7.1: Non-electric device to provide mechanotactile feedback, developed byAntfolk et al. [51].

stimulation operates force application on the skin. It is most often combined witha phantom hand map used as target for sensory feedback. It is a technique appeal-ing because of its simplicity and e�ectiveness. Antfolk et al., investigated furtherthis idea developing a non-electric device (Fig. 7.1) and proving its e�cacy with 12amputee test subjects [51].

Direct neural stimulation is the most exciting approach for sensory feedback, sinceit can potentially restore somatic sensory de�ciencies. For this reason it has gatheredmany attentions from researchers around the world. The �rst attempts on using im-planted electrodes to restore sensory feedback were conducted over 40 years ago [53],and several others were reported more recently [54�56]. Unfortunately, a more sys-tematic knowledge of the interactions between the peripheral and central nervoussystems is imperative and still missing. Moreover, current technology limitations inmatter of neural interfaces prevent amputee patients to exploit their daily roboticprostheses via �natural feeling� closed-loop control. The poor selectivity of neuralinterfaces is one of the major limitations. Furthermore, the selectivity is inverselyproportional to the functional duration of the implanted electrodes; the most promis-ing studies in recent literature are carried in controlled laboratory environments forseveral months, after which the implants are removed.

24

Chapter 8Summary of the Thesis Contributions

In the last part of this report are appended four scienti�c articles written within thetime of this thesis, three of them are already published and the fourth is currentlyunder reviewing process. A brief description of their contributions is presented inwhat follows.

• Paper I presents tests and considerations needed during the design phase ofthe embedded system aim of this thesis. It focuses on the identi�cation of themost suitable analog front-end for EMG signal acquisition in the context ofa low-power and low-cost prosthetic controller. Two popular analog front-endchips available on the market were analyzed and compared. Preliminary testsare also included regarding its feasibility for myoelectric pattern recognitionapplied to prosthetic control. These considerations might be bene�cial to otherresearchers and developers in the �eld of prosthetic.

• Paper II describes a low-cost and open-source bioelectric signals acquisitionarchitecture. It was designed as a modular system where each single unit canprovide up to 8 di�erential or single-ended channels with a resolution of 24-bitsand reprogrammable gain up to 24 V/V. It was meant as a hardware comple-mentary to an open-source software platform already available online from ourgroup, namely BioPatRec. This integration provide a complete system for re-searchers interested on intuitive prosthetic control based on pattern recognitionalgorithms.

• Paper III reports the development and the functional validation of the embed-ded system designed to exploit the advantages of the OHMG technology. TheArti�cial Limb Controller allows for bioelectric signals acquisition, processingand decoding of motor intent towards prosthetic control. It includes a neu-rostimulator to provide direct neural feedback thus enabling closed-loop roboticlimb control. The system was validated and its functionality was proven in a

25

Chapter 8. Summary of the Thesis Contributions

�rst pilot OHMG-patient. This embedded controller allows for out-of-the-labapplications of closed-loop prosthetic control.

• Paper IV presents the report of a short-term clinical application of the devel-oped prosthetic controller (presented in Paper III) utilized in combination withnon-invasive EMG electrodes and pattern recognition methods. The intuitive-control approach allowed for the discrimination of three �ne grips and open/closehand in a multifunctional prosthetic hand. The system was used by a dysmeliasubject for �ve consecutive days in a out-of-the-lab context while informationabout prosthesis usage and real-time classi�cation accuracy were collected. Thefunctionality of the proposed approach was compared with the conventionalmyoelectric control approach. This work presents an alternative to the conven-tional use of myoelectric signals in combination with multifunctional prosthetichands, moreover it is also a further validation of the Arti�cial Limb Controller.

26

Chapter 9General Conclusions and Future Work

Some of the most exciting advancements in the �eld of upper limb prosthetic fromall around the world have been introduced within this thesis, with its main focus seton intuitive myoelectric prostheses, which, fed by �natural� muscles' activity on thestump, can trigger movements on the prosthesis. Promising developments are stillongoing worldwide, moreover the popularity of the prosthetic �eld is growing rapidly,perhaps dragged by the science-�ction-biased appeal of robotics. Unfortunately forpatients, clinical implementations still lag behind the advancements of research inmatter of robotic limbs, prosthetic control and sensory feedback. More e�orts aretherefore needed to �ll the gap.

Within this work, it was developed an embedded system capable of prostheticcontrol via the processing of bioelectric signals with pattern recognition algorithms.The Arti�cial Limb Controller includes a neurostimulator to provide direct neuralfeedback based on sensory information. The system was ultimately designed to bereliably used in activities of daily living for real clinical implementations, as well as aresearch platform to monitor prosthesis usage and training, machine learning basedcontrol algorithms, and neural stimulation paradigms. It was shown that it can beused in combination with the implanted electrodes on nerves and muscles providedby the OHMG technology.

However, considering the early clinical test stage of the OHMG implant, restrictedinclusion criteria and implant costs preclude a massive distribution of this solution.For this reason, many e�orts of our Biomechatronics and Rehabilitation Laboratoryare still addressed to alternative and more accessible solutions, such as less invasivesensors (surface EMG electrodes). Therefore, a clinical proof-of-concept in activitiesof the daily life was conducted, by designing a complete prosthetic system based onthe Arti�cial Limb Controller. The system was used by a dysmelia subject for �vedays for the natural control of three �ne grips and open/close in a multi-functionalprosthetic hand.

27

Chapter 9. General Conclusions and Future Work

A low-cost hardware for bioelectric signal acquisition was designed and sharedwith the scienti�c community. It is a complete open-source package, comprisinghardware and software, made freely accessible via GitHub platform. This integrationprovide a complete system for intuitive myoelectric control where signal processing,machine learning, and control algorithms are used for the prediction of motor volitionand the control of robotic and virtual prostheses.

This thesis work provided the hardware (Arti�cial Limb Controller) needed forfurther investigations in many directions. The ultimate focus is set on the intuitivecontrol of a prosthetic limb. We aim for a control interface that is "as natural as it canbe", where the user is relieved from the burden of training and adaptation. We believethat the combination of the OHMG technology and the TMR procedure, strengthenedby the advancements of signal processing techniques, can very well improve the statusquo for amputees in matter of functional control of their prosthetic limbs. We trulybelieve in the advantages of osseointegration and in the breakthrough represented bythe OHMG technology, it is now mandatory to deliver further assessments of theirbene�ts to the community. We think that sensory feedback should be an essentialrequirement of any modern upper limb prosthesis, and it is now a concrete clinicalpossibility thanks to the long-term stable access to nerves via the OHMG. The sameaccess channel on the nerves is going to be also used to investigate the chances of usingnerves' signals for control purposes. More intelligent signal processing algorithms canbe studied to optimize the control, as well as their feasibility for clinical applications.These are some of the ideas and directions that are likely to be taken in the followingtime of this industrial doctorate project.

28

References

[1] M. Ortiz-Catalan, B. Hakansson, and R. Branemark, �An osseointegratedhuman-machine gateway for long-term sensory feedback and motor controlof arti�cial limbs,� Science Translational Medicine, vol. 6, no. 257, pp.257re6�257re6, oct 2014. [Online]. Available: http://stm.sciencemag.org/cgi/doi/10.1126/scitranslmed.3008933

[2] K. J. Zuo and J. L. Olson, �The evolution of functional hand replacement: Fromiron prostheses to hand transplantation,� Canadian Journal of Plastic Surgery,vol. 22, no. 1, pp. 44�51, 2014.

[3] K. Østlie, I. M. Lesjø, R. J. Franklin, B. Garfelt, O. H. Skjeldal, and P. Magnus,�Prosthesis use in adult acquired major upper-limb amputees: patterns of wear,prosthetic skills and the actual use of prostheses in activities of daily life.�Disability and Rehabilitation: Assistive Technology, vol. 7, no. 6, pp. 479�93,2012. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/22315926

[4] C. J. De Luca, �Control of upper-limb prostheses: a case for neuroelectric con-trol.� pp. 57�61, 1978.

[5] J. Billock, �Upper limb prosthetic terminal devices: Hands versus hooks,� Clin-ical Prosthetics and Orthotics, vol. 10, no. 2, pp. 57�65, 1986.

[6] T. a. Kuiken, G. a. Dumanian, R. D. Lipshutz, L. a. Miller, and K. a. Stub-ble�eld, �The use of targeted muscle reinnervation for improved myoelectricprosthesis control in a bilateral shoulder disarticulation amputee,� Prostheticsand Orthotics International, vol. 28, pp. 245�253, 2004.

[7] T. Kuiken, G. Li, B. A. Lock, R. D. Lipschutz, L. A. Miller, K. A. Stubble�eld,and K. B. Englehart, �Targeted muscle reinnervation for real-time myoelectriccontrol of multifunction arti�cial arms,� JAMA: the journal of the AmericanMedical Association, vol. 301, no. 6, pp. 619�628, 2009. [Online]. Available:http://jama.ama-assn.org/content/301/6/619.short

29

References

[8] J. S. Hebert, J. L. Olson, M. J. Morhart, M. R. Dawson, P. D. Marasco, T. A.Kuiken, and K. M. Chan, �Novel targeted sensory reinnervation technique torestore functional hand sensation after transhumeral amputation,� IEEE Trans-actions on Neural Systems and Rehabilitation Engineering, vol. 22, no. 4, pp.765�773, 2014.

[9] T. a. Kung, R. a. Bueno, G. K. Alkhalefah, N. B. Langhals, M. G. Urbanchek,and P. S. Cederna, �Innovations in prosthetic interfaces for the upper extremity.�Plastic and reconstructive surgery, vol. 132, no. 6, pp. 1515�1523, 2013.

[10] M. G. Urbanchek, T. A. Kung, C. M. Frost, D. C. Martin, L. M. Larkin, A. Woll-stein, and P. S. Cederna, �Development of a Regenerative Peripheral Nerve In-terface for Control of a Neuroprosthetic Limb,� BioMed Research International,vol. 2016, 2016.

[11] T. a. Kung, N. B. Langhals, D. C. Martin, P. J. Johnson, P. S.Cederna, and M. G. Urbanchek, �Regenerative peripheral nerve interfaceviability and signal transduction with an implanted electrode.� Plastic andreconstructive surgery, vol. 133, no. 6, pp. 1380�94, 2014. [Online]. Available:http://www.ncbi.nlm.nih.gov/pubmed/24867721

[12] J. T. Shores, V. Malek, W. P. A. Lee, and G. Brandacher, �Outcomes after handand upper extremity transplantation,� Journal of Materials Science: Materialsin Medicine, vol. 28, no. 5, 2017.

[13] P. Brånemark, �Osseointegrated implants in the treatment of the edentulous jaw.Experience from 10 year period,� Scandinavian Journal of Plastic and Recon-structive Surgery and Hand Surgery, vol. 16, no. 1, pp. 1�32, 1977.

[14] K. Hagberg, R. Brånemark, B. Gunterberg, and B. Rydevik, �Osseointegratedtrans-femoral amputation prostheses: prospective results of general andcondition-speci�c quality of life in 18 patients at 2-year follow-up.� Prostheticsand orthotics international, vol. 32, no. 1, pp. 29�41, 2008. [Online]. Available:http://www.ncbi.nlm.nih.gov/pubmed/18330803

[15] H. Van De Meent, M. T. Hopman, and J. P. Frölke, �Walking ability and qualityof life in subjects with transfemoral amputation: A comparison of osseointegra-tion with socket prostheses,� Archives of Physical Medicine and Rehabilitation,vol. 94, no. 11, pp. 2174�2178, 2013.

[16] R. Brånemark, Ö. Berlin, K. Hagberg, P. Bergh, B. Gunterberg, andB. Rydevik, �A novel osseointegrated, percutaneous prosthetic system fortreatment of patients with transfemoral amputation: A prospective study of51 patients,� The Bone & Joint Journal, vol. 96-B, no. 1, pp. 106�113, 2014.[Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/24395320

30

References

[17] K. Hagberg and R. Brånemark, �One hundred patients treated with osseointe-grated transfemoral amputation prostheses�rehabilitation perspective.� Journalof rehabilitation research and development, vol. 46, no. 3, pp. 331�344, 2009.

[18] M. S. Johannes, J. D. Bigelow, J. M. Burck, S. D. Harshbarger, M. V. Kozlowski,and T. Van Doren, �An overview of the developmental process for the modularprosthetic limb,� Johns Hopkins APL Technical Digest (Applied Physics Labo-ratory), vol. 30, no. 3, pp. 207�216, 2011.

[19] L. Resnik, S. L. Klinger, and K. Etter, �The DEKA Arm: its features,functionality, and evolution during the Veterans A�airs Study to optimize theDEKA Arm.� Prosthetics and orthotics international, vol. 38, no. 6, pp. 492�504,2014. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/24150930

[20] A. Roche, H. Rehbaum, D. Farina, and O. Aszmann, �Pros-thetic Myoelectric Control Strategies: A Clinical Perspective,� Cur-rent Surgery Reports, vol. 2, no. 3, pp. 1�11, 2014. [Online].Available: http://link.springer.com/article/10.1007/s40137-013-0044-8{\%}5Cnhttp://link.springer.com/content/pdf/10.1007/s40137-013-0044-8.pdf

[21] K. Østlie, I. M. Lesjø, R. J. Franklin, B. Garfelt, O. H. Skjeldal, and P. Mag-nus, �Prosthesis rejection in acquired major upper-limb amputees: a population-based survey,� Disability and Rehabilitation: Assistive Technology, vol. 7, no. 4,pp. 294�303, 2012.

[22] R. W. Wirta, D. R. Taylor, and F. R. Finley, �Pattern-recognition arm prosthe-sis: a historical perspective-a �nal report.� Bulletin of prosthetics research, pp.8�35, 1978.

[23] P. Herberts, C. Almström, R. Kadefors, and P. D. Lawrence, �Hand prosthesiscontrol via myoelectric patterns.� Acta orthopaedica Scandinavica, vol. 44, no. 4,pp. 389�409, 1973.

[24] D. Graupe, W. J. Monlux, and I. Magnussen, �A multifunctional prosthesiscontrol system based on time series identi�cation of emg signals using micropro-cessors,� Bulletin of Prosthetics Research, vol. 101, no. 134, 1977.

[25] E. J. Scheme and K. Englehart, �Electromyogram pattern recognition forcontrol of powered upper-limb prostheses: State of the art and challenges forclinical use,� J Rehabil Res Dev, vol. 48, no. 6, p. 643, 2011. [Online]. Available:http://www.rehab.research.va.gov/jour/11/486/pdf/scheme486.pdf

[26] D. Farina, N. Jiang, H. Rehbaum, A. Holobar, B. Graimann, H. Dietl, andO. C. Aszmann, �The extraction of neural information from the surface EMG

31

References

for the control of upper-limb prostheses: Emerging avenues and challenges,�IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 22,no. 4, pp. 797�809, 2014.

[27] M. Atzori, M. Cognolato, and H. Müller, �Deep learning with convolutional neu-ral networks applied to electromyography data: A resource for the classi�cationof movements for prosthetic hands,� Frontiers in Neurorobotics, vol. 10, no. SEP,2016.

[28] G. Ghazaei, A. Alameer, P. Degenaar, G. Morgan, and K. Nazarpour,�Deep learning-based arti�cial vision for grasp classi�cation in myoelectrichands,� Journal of Neural Engineering, vol. 14, no. 3, p. 036025, 2017.[Online]. Available: http://stacks.iop.org/1741-2552/14/i=3/a=036025?key=crossref.93a9e8995f57f4167ed43697c326f406

[29] M. Yamada, N. Niwa, and A. Uchiyama, �Multifunctional hand prosthesiscontrol methods using EMG signals,� Japanese Journal of Medical Electronicsand Biological Engineering, vol. 18, no. 2, pp. 133�138, 1980. [Online]. Avail-able: https://www.scopus.com/inward/record.uri?eid=2-s2.0-0019277879{\&}partnerID=40{\&}md5=3f8241a217af6149334ca0d550154049

[30] A. Soares, A. Andrade, E. Lamounier, and R. Carrijo, �The development of avirtual myoelectric prosthesis controlled by an EMG pattern recognition systembased on neural networks,� Journal of Intelligent Information Systems, vol. 21,no. 2, pp. 127�141, 2003.

[31] M. Ortiz-Catalan, R. Brånemark, and B. Håkansson, �BioPatRec: A modularresearch platform for the control of arti�cial limbs based on pattern recognitionalgorithms.� Source code for biology and medicine, vol. 8, p. 11, 2013. [Online].Available: https://github.com/biopatrec/biopatrec/wiki

[32] M. R. Dawson, J. P. Carey, and F. Fahimi, �Myoelectric training systems.�Expert review of medical devices, vol. 8, no. 5, pp. 581�589, 2011. [Online].Available: http://www.ncbi.nlm.nih.gov/pubmed/22026623

[33] M. Ortiz-Catalan, R. A. Gudmundsdottir, M. B. Kristo�ersen, A. Zepeda-Echavarria, K. Caine-Winterberger, K. Kulbacka-Ortiz, C. Widehammar,K. Eriksson, A. Stockselius, C. Ragn??, Z. Pihlar, H. Burger, and L. Hermans-son, �Phantom motor execution facilitated by machine learning and augmentedreality as treatment for phantom limb pain: a single group, clinical trial in pa-tients with chronic intractable phantom limb pain,� The Lancet, vol. 388, no.10062, pp. 2885�2894, 2016.

32

References

[34] W. Xiao, H. Huang, Y. Sun, and Q. Yang, �Promise of embedded system withGPU in arti�cial leg control: Enabling time-frequency feature extraction fromelectromyography,� in Proceedings of the 31st Annual International Conferenceof the IEEE Engineering in Medicine and Biology Society: Engineering the Fu-ture of Biomedicine, EMBC 2009, 2009, pp. 6926�6929.

[35] A. Boschmann, A. Agne, L. Witschen, G. Thombansen, F. Kraus, andM. Platzner, �FPGA-based Acceleration of High Density Myoelectric Signal Pro-cessing,� Proceedings of International Conference on ReConFigurable Computingand FPGAs (ReConFig), 2015.

[36] G. C. Chang, W. J. Kang, L. Jer-Junn, C. K. Cheng, J. S. Lai, J. J. J. Chen, andT. S. Kuo, �Real-time implementation of electromyogram pattern recognition asa control command of man-machine interface,� Medical Engineering and Physics,vol. 18, no. 7, pp. 529�537, 1996.

[37] F. Tenore, R. S. Armiger, R. J. Vogelstein, D. S. Wenstrand, S. D. Harshbarger,and K. Englehart, �An embedded controller for a 7-degree of freedom prostheticarm,� Proceedings of the Annual International Conference of the IEEE Engi-neering in Medicine and Biology, vol. 2008, pp. 185�188, 2008.

[38] C.-L. Lin, S.-C. Wang, H.-C. Wu, S.-T. Young, M.-H. Lee, and T.-S. Kuo, �Aspeech controlled arti�cial limb based on DSP chip,� Proceedings of the AnnualInternational Conference of the IEEE Engineering in Medicine and Biology So-ciety, vol. 5, no. 5, pp. 2704�2705, 1998.

[39] T. Hirata, T. Nakamura, R. Kato, S. Morishita, and H. Yokoi, �Development ofmobile controller for EMG prosthetic hand with tactile feedback,� IEEE/ASMEInternational Conference on Advanced Intelligent Mechatronics, AIM, pp. 110�115, 2011.

[40] H. Liu, D. P. Yang, L. Jiang, and S. W. Fan, �Development of a multi-DOFprosthetic hand with intrinsic actuation, intuitive control and sensory feedback,�Industrial Robot-an International Journal, vol. 41, no. 4, pp. 381�392, 2014.

[41] S. Benatti, F. Casamassima, B. Milosevic, E. Farella, P. Schonle, S. Fateh,T. Burger, Q. Huang, and L. Benini, �A Versatile Embedded Platform forEMG Acquisition and Gesture Recognition.� IEEE transactions on biomedicalcircuits and systems, vol. PP, no. 99, p. 1, 2015. [Online]. Available:http://www.ncbi.nlm.nih.gov/pubmed/26513799

[42] J. a. Ho�er and G. E. Loeb, �Implantable electrical and mechanical interfaceswith nerve and muscle.� Annals of biomedical engineering, vol. 8, pp. 351�360,1980.

33

References

[43] D. Tan, M. Schiefer, M. W. Keith, R. Anderson, and D. J. Tyler, �Stability andselectivity of a chronic , multi-contact cu� electrode for sensory stimulation ina human amputee,� in 6th Ann Int IEEE/EMBS Conf Neural Eng, San Diego,nov 2013, pp. 859�862.

[44] P. Herberts, R. Kadefors, E. Kaiser, and I. Petersén, �Implantationof micro-circuits for myo-electric control of prostheses,� The Bone &Joint Journal, vol. 50, no. 4, pp. 780�91, 1968. [Online]. Available:http://www.ncbi.nlm.nih.gov/pubmed/5706883

[45] F. Clippinger, R. Avery, and B. Titus, �A sensory feedback system for anupper-limb amputation prosthesis,� Bulletin of prosthetics research, pp. 247�258,1974. [Online]. Available: http://ukpmc.ac.uk/abstract/MED/4462906

[46] P. F. Pasquina, M. Evangelista, A. J. Carvalho, J. Lockhart, S. Gri�n, G. Nanos,P. McKay, M. Hansen, D. Ipsen, J. Vandersea, J. Butkus, M. Miller, I. Murphy,and D. Hankin, �First-in-man demonstration of a fully implanted myoelectricsensors system to control an advanced electromechanical prosthetic hand,� Jour-nal of Neuroscience Methods, vol. 244, pp. 85�93, 2015.

[47] R. F. Weir, �Implantable Myoelectric Sensors (IMEs) for intramuscolar Elec-tromyogram recording,� IEEE Transactions on Biomedical Engineering, vol. 29,no. 6, pp. 997�1003, 2009.

[48] P. J. Kyberd, C. Wartenberg, L. Sandsjo, S. Jonsson, D. Gow, J. Frid, C. Alm-strom, and L. Sperling, �Survey of Upper-Extremity Prosthesis Users in Swedenand the United Kingdom,� Journal of Prosthetics and Orthotics, vol. 19, no. 2,pp. 55�62, 2007.

[49] E. Biddiss, D. Beaton, and T. Chau, �Consumer design prioritiesfor upper limb prosthetics,� Disability and Rehabilitation: AssistiveTechnology, vol. 2, no. 6, pp. 346�357, 2007. [Online]. Available: http://www.tandfonline.com/doi/abs/10.1080/17483100701714733

[50] C. Antfolk, M. D'Alonzo, B. Rosén, G. Lundborg, F. Sebelius, andC. Cipriani, �Sensory feedback in upper limb prosthetics.� Expert reviewof medical devices, vol. 10, no. 1, pp. 45�54, 2013. [Online]. Available:http://www.ncbi.nlm.nih.gov/pubmed/23278223

[51] C. Antfolk, A. Björkman, S. O. Frank, F. Sebelius, G. Lundborg, and B. Rosen,�Sensory feedback from a prosthetic hand based on airmediate d pressure fromthe hand to the forearm skin,� Journal of Rehabilitation Medicine, vol. 44, no. 8,pp. 702�707, 2012.

34

References

[52] K. A. Kaczmarek, J. G. Webster, P. Bach-y Rita, and W. J. Tompkins, �Elec-trotactile and Vibrotactile Displays for Sensory Substitution Systems,� IEEETransactions on Biomedical Engineering, vol. 38, no. 1, pp. 1�16, 1991.

[53] F. Clippinger, R. Avery, and B. Titus, �A sensory feedback system for anupper-limb amputation prosthesis.� Bulletin of Prosthetics Research, vol. 10-22,pp. 247�258, 1974. [Online]. Available: http://europepmc.org/abstract/MED/4462906

[54] C. M. Oddo, S. Raspopovic, F. Artoni, A. Mazzoni, G. Spigler, F. Petrini,F. Giambattistelli, F. Vecchio, F. Miraglia, L. Zollo, G. Di Pino, D. Camboni,M. C. Carrozza, E. Guglielmelli, P. M. Rossini, U. Faraguna, and S. Micera,�Intraneural stimulation elicits discrimination of textural features by arti�cial�ngertip in intact and amputee humans,� eLife, vol. 5, no. MARCH2016, 2016.

[55] T. S. Davis, H. A. C. Wark, D. T. Hutchinson, D. J. Warren, K. O'Neill,T. Scheinblum, G. A. Clark, R. A. Normann, and B. Greger, �Restoring motorcontrol and sensory feedback in people with upper extremity amputations usingarrays of 96 microelectrodes implanted in the median and ulnar nerves.� Journalof neural engineering, vol. 13, no. 3, p. 036001, 2016. [Online]. Available:http://www.ncbi.nlm.nih.gov/pubmed/27001946

[56] D. W. Tan, M. A. Schiefer, M. W. Keith, J. R. Anderson, J. Tyler, and D. J.Tyler, �A neural interface provides long-term stable natural touch perception,�Science Translational Medicine, vol. 6, no. 257, 2014.

35