robotic software frameworks and software component models in the development of automated handling...

17
RESEARCH PAPER Robotic software frameworks and software component models in the development of automated handling of individual natural fibers Mathias von Essen & Juha Hirvonen & Seppo Kuikka & Pasi Kallio Received: 31 December 2013 /Revised: 31 May 2014 /Accepted: 4 June 2014 # Springer-Verlag Berlin Heidelberg 2014 Abstract Automated micro- and nanohandling systems are highly dependent on the robustness and quality of their control software. However, aspects related to software are rarely discussed in this context. This paper describes how mature robotic software frameworks and software component models can be applied on the software development of micro- and nanohandling applications. The applicability of the selected approach is demonstrated by developing component-based software that relies on Robotic Construction Kit, which is a mature robotic software framework. The demonstration con- centrates on image-based pick-and-place procedures for the manipulation of individual natural fibers. The developed pro- cedures are a part of an effort to develop a fully automated high throughput instrument for analyzing mechanical proper- ties at a fiber level. The pick-and-place sequences are divided into reusable unit operations that can be utilized to implement different tasks which the instrument must perform. Two dif- ferent demonstrations: pick and place, and coordinated fiber lifting are presented. The applicability of the selected software development approach is evaluated with two criteria: the execution time and the success rate of the demonstrated oper- ations. The developed automated coordinated fiber lifting sequence is compared against its teleoperated counterpart. The success rates were 80 and 82.5 %, respectively for teleoperated and automated sequences. The execution time of automated sequence was 51.5 s on average. The teleoperated reference experiment had significantly longer execution time, being 131 s on average. Keywords Microrobotics . Microhandling . Micromanipulation . Automation . Robotic software framework . Software component models 1 Introduction Micro- and nanorobotic technologies experienced a rapid evolution during the past decade and they have enabled the research and assembly of objects that is out of reach for conventional techniques. The technological maturity has reached a sufficient level for commercial applications. How- ever, the breakthrough of micro-and nanorobotic systems in industrial scale applications has not been seen. Before micro- and nanorobotic systems have the possibility to succeed in industrial applications requiring high throughput, the level of automation must increase, i.e. development of automated or even autonomous microrobotic systems is essential. One of the potential high throughput industrial applications of microhandling is the characterization of natural fibrous materials that are construction units for a broad spectrum of products. The material properties of fiber-based products are largely determined by the properties of the underlying fiber network and its two main components: the fibers and the bonds between the fibers. Despite this central role of fibers, their research still has many shortcomings. One of the main drawbacks is the lack of reliable measurement data concerning especially the fiber-fiber bonds and fiber networks, but also the characteristics of individual fibers. At a single fiber level, mechanical properties that contribute to the quality of the end product include parameters such as stiffness, tensile strength, and compressibility. Automated micro- and nanohandling systems are highly dependent on the robustness and quality of their control soft- ware. However, aspects related to software are rarely M. von Essen (*) : J. Hirvonen : S. Kuikka : P. Kallio Department of Automation Science and Engineering, Tampere University of Technology, Tampere, Finland e-mail: [email protected] J Micro-Bio Robot DOI 10.1007/s12213-014-0078-8

Upload: pasi

Post on 25-Jan-2017

215 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

RESEARCH PAPER

Robotic software frameworks and software component modelsin the development of automated handling of individualnatural fibers

Mathias von Essen & Juha Hirvonen & Seppo Kuikka &

Pasi Kallio

Received: 31 December 2013 /Revised: 31 May 2014 /Accepted: 4 June 2014# Springer-Verlag Berlin Heidelberg 2014

Abstract Automated micro- and nanohandling systems arehighly dependent on the robustness and quality of their controlsoftware. However, aspects related to software are rarelydiscussed in this context. This paper describes how maturerobotic software frameworks and software component modelscan be applied on the software development of micro- andnanohandling applications. The applicability of the selectedapproach is demonstrated by developing component-basedsoftware that relies on Robotic Construction Kit, which is amature robotic software framework. The demonstration con-centrates on image-based pick-and-place procedures for themanipulation of individual natural fibers. The developed pro-cedures are a part of an effort to develop a fully automatedhigh throughput instrument for analyzing mechanical proper-ties at a fiber level. The pick-and-place sequences are dividedinto reusable unit operations that can be utilized to implementdifferent tasks which the instrument must perform. Two dif-ferent demonstrations: pick and place, and coordinated fiberlifting are presented. The applicability of the selected softwaredevelopment approach is evaluated with two criteria: theexecution time and the success rate of the demonstrated oper-ations. The developed automated coordinated fiber liftingsequence is compared against its teleoperated counterpart.The success rates were 80 and 82.5 %, respectively forteleoperated and automated sequences. The execution timeof automated sequence was 51.5 s on average. Theteleoperated reference experiment had significantly longerexecution time, being 131 s on average.

Keywords Microrobotics .Microhandling .

Micromanipulation . Automation . Robotic softwareframework . Software component models

1 Introduction

Micro- and nanorobotic technologies experienced a rapidevolution during the past decade and they have enabled theresearch and assembly of objects that is out of reach forconventional techniques. The technological maturity hasreached a sufficient level for commercial applications. How-ever, the breakthrough of micro-and nanorobotic systems inindustrial scale applications has not been seen. Before micro-and nanorobotic systems have the possibility to succeed inindustrial applications requiring high throughput, the level ofautomation must increase, i.e. development of automated oreven autonomous microrobotic systems is essential.

One of the potential high throughput industrial applicationsof microhandling is the characterization of natural fibrousmaterials that are construction units for a broad spectrum ofproducts. The material properties of fiber-based products arelargely determined by the properties of the underlying fibernetwork and its two main components: the fibers and thebonds between the fibers. Despite this central role of fibers,their research still has many shortcomings. One of the maindrawbacks is the lack of reliable measurement data concerningespecially the fiber-fiber bonds and fiber networks, but alsothe characteristics of individual fibers. At a single fiber level,mechanical properties that contribute to the quality of the endproduct include parameters such as stiffness, tensile strength,and compressibility.

Automated micro- and nanohandling systems are highlydependent on the robustness and quality of their control soft-ware. However, aspects related to software are rarely

M. von Essen (*) : J. Hirvonen : S. Kuikka : P. KallioDepartment of Automation Science and Engineering, TampereUniversity of Technology, Tampere, Finlande-mail: [email protected]

J Micro-Bio RobotDOI 10.1007/s12213-014-0078-8

Page 2: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

discussed in this context. This paper proposes the usage ofrobotic software frameworks, and software componentmodels in the field of micro- and nanohandling. The objectiveis to evaluate how existing well-established robotic softwareframeworks and component models can be applied in micro-and nanorobotic applications. The motivation is to promotereusable and agile methodology where micro- andnanohandling tasks are implemented in the software as a setof functional elements that are required by a particular task.The selected software development approach is demonstratedby implementing one of the key operations in all microroboticwood fiber characterization tasks on an existingmicrohandling platform. The key operation implemented inthis paper is a pick and place operation of an individual woodfiber. The demonstration focuses on the handling of woodfibers, but the described approach is also applicable to othernatural fibrous materials of the same scale. The applicabilityof the selected software development approach is evaluatedwith two criteria: the execution time and the success rate of thepick and place operation. An automated pick and place oper-ation that is implemented on a selected robotic softwareframework, is compared with a teleoperated counterpart usingthe selected performance criteria.

1.1 Microrobotics in pulp and paper research

In pulp and paper research, the current methods for determin-ing material properties are mainly based on sheet level mea-surements. However, there is an increasing interest to obtaininformation on single fiber level. Paper is a fundamentally anetwork structure of different types of wood cells, mainlyfibers and fines. The dimensions of fibers vary from 0.8–4.5 mm in length and the diameters are in the range of 16–70 μm [1]. Understanding about the mechanical properties ofthe individual wood fibers could lead to significant improve-ments in the development of wood fiber-based products.

The advances of microrobotics and microsystems technol-ogy have provided new tools and methods to manipulate andcharacterize micro- and nanoscale samples such as living cells[2–8], carbon nanotubes [9,10], and to promotemicroassembly approaches [11–14]. In fiber research, theaforementioned technologies enable the characterization ofindividual fibers and fiber-fiber bonds. The methodologyprovides an attractive alternative to conventional characteri-zation procedures, as the obtained results are not coupled withundesired information. The potential of microrobotic charac-terization in wood fiber research has been exhibited in[15–21], where several different teleoperated fiber level char-acterization procedures, including (i) the flexibility measure-ment of individual fibers and (ii) the force measurement ofindividual fiber-fiber bonds, have been reported.Microrobotics can also be utilized for the preparation offibrous samples for other tools, such as Scanning Electron

Microscopy (SEM), x-ray tomography, and Atomic ForceMicroscopy (AFM). Currently, the preparation of individualfiber samples in pulp and paper studies is performedmanually.Automatic sample preparation could significantly improve thethroughput of fiber-level analyses due to the reduced timeconsumption. Furthermore, the quality of the prepared sam-ples can be improved due to more accurate positioning and thereduced risk of cross contamination.

In order to fully utilize the benefits of the presented fiber-level characterization capabilities, the described proceduresmust be automated. Handling of individual fibers requiresdexterous high-precision operations, which are often beyondhuman capabilities. High throughput is essential as thousandsof fibers must be characterized to produce statistically reliabledata. The key difficulties in the microhandling of wood fibersare related to the heterogeneity and complex geometry of thestudied objects; wood fibers have a high variance in bothlength and diameter. Further, they twist and curl in all threedimensions, which poses challenges imaging and selection ofmanipulation strategy.

1.2 Software aspects in microrobotic applications

Complex automated systems are highly dependent on therobustness and quality of their control software. However,the important role of software is often neglected in the devel-opment of microrobotic systems. In a typical case, the soft-ware is interfacing in-house built custom hardware to performa specific task. This often results in a custom-designed mono-lithic software architecture that is incapable of accommodat-ing to changes in the hardware configuration or in the per-formed task. As a consequence, a vast number of the devel-oped robot functionality, algorithms and control schemes can-not be reused as such, but need to be completely rewritten foreach application. The constant reinvention of the alreadyexisting functionality leads not only to a huge waste of re-sources, but also hinders the development of robust sourcecode.

One approach to develop modular and reusable sourcecode for microhandling applications is to divide the performedmanipulation tasks into unit operations. The designed controlsoftware should then implement and allow the integration ofthe unit operations in such way that the system integrator cancreate different sequences from existing unit operationimplementations. The goal is that recurring usage of any unitfunction should minimize the need for writing additionalsource code. Moreover, a developer who wishes to reuseexisting unit operations should not be required to have in-depth knowledge of the unit operations’ implementation.Rather, the unit operations should be considered as black-box models whose behavior and interfaces are well document-ed. Such an approach would also enable the sharing of ready-made unit functions between research groups.

J Micro-Bio Robot

Page 3: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

From software engineering perspective, Component-BasedSoftware Engineering (CBSE) is a suitable paradigm to coverthe described needs. In software engineering, a component isa unit that encapsulates specific functionality. In the roboticsdomain, the functionality could be, for example, an algorithm,a control paradigm or a device driver. However, CBSE ismainly a detailed design and implementation approach andas such does not take a stance on generic architectural aspectssuch as distribution or scalability. These supportive, but nec-essary aspects are responsibilities of a software framework.

A typical robotic software framework (RSF) provides acomponent model, communication middleware (usuallythird-party solution, such as CORBA1) and mechanisms tomanage the state and the life cycle of the components. Inindustrial and service robotics, the growing need to reusesource code implementing robotic algorithms and devicedrivers has resulted in the development of multiple roboticsoftware frameworks (RSFs). RSFs target at typical roboticproblems such as navigation, motion planning, real-time con-trol, distribution, mobility within the robotics, or the combi-nations of the aforementioned.

Communication between components typically utilizes theconcepts of connections and ports. Ports are utilized as ameans of exposing component’s interface, so that other com-ponents can access the functionality of the component. Con-nections provide abstraction for communication between theports. Typical RSFs provide at least two types of ports; a dataflow port and a service port. The first one is utilized in one-way communication, where data flows from one componentto another, e.g. data from a sensor component flows to acontroller component. The service port implements request-response type of communication, where one component callsa function of another component.

Well known open-source RSFs targeting at industrial andservice robotics include Player [22], Orca [23], Orocos [24],Robotic Operating System (ROS) [25], Yet Another RoboticPlatform (YARP) [26], Open Robotic TechnologyMiddleware (OpenRTM-aist) [27], Open Platform for RoboticServices (OPRoS) [28], Open Robotic Development Kit(OpenRDK) [29], SmartSoft [30] and Robotic ConstructionKit (Rock) [31].

Even though microrobotic applications have their owndistinctive characteristics when compared with industrial andservice robotics, the existing RSFs provide significant benefitsthrough reduced development time and solid code base. Forexample, the construction of complex real-time systems isextremely time consuming and requires thorough knowledgeof the operating system, networking, etc. It also requiresintensive testing and validation. In such cases, the

development cycle can be significantly shortened by utilizingan existing RSF.

The first steps towards automated fiber manipulation werereported in [18]. In this paper, we continue the effort towards afully automated fiber characterization platform by demonstrat-ing two automated procedures that are useful in fiber charac-terization: pick and place; and coordinated lifting. The firstprocedure is utilized to move individual fibers to a desiredlocation. The latter procedure is required in the breaking offiber-fiber bonds. The procedures are divided into unit oper-ations which are then encapsulated into separate softwarecomponents. The control software presented in this work aimsto promote high reusability by applying CBSE. It relies onRock, which is an RSF designed for real-time and distributedrobot and machine control. Rock is closely related withOrocos; Rock utilizes the component model developed inOrocos. The main difference between the two is that Rockhas a larger variety of off-the-shelf components and graphicalintrospection tools. The rest of this paper is organized asfollows. Section 2 presents the unit operations and two char-acterization procedures that utilize the unit operations. Sec-tion 3 describes the experimental setup while Section 4 isdedicated to the control software. Section 5 presents theresults of the automated pick and place, and the coordinatedlifting procedures. Section 6 concludes the paper.

2 Unit operations and fiber characterization procedures

Fiber characterization procedures carried out with the devel-oped platform consist of a set of unit operations. In this work,we present the following unit operations: Detect, Grasp,Move, Pull, and Release. Two fiber characterization proce-dures, Fiber Pick and Place (FPP), and Coordinated FiberLifting (CFL), are presented as the examples of sequencesthat can be created from the described unit operations. Dem-onstration of the pick and place and coordinated lifting pro-cedures are provided in Section 5.

2.1 Unit operations

Most of the developed unit operations rely on the image-baseddetection of the fiber or the fiber-fiber bond to bemanipulated.The Detect operation is an image-based unit operation whichaims to locate and select a fiber that is suitable for character-ization. Suitability is determinedwith a set of parameters, suchas the curliness and the length of the fiber. Moreover, the fibershould not be entangled with other fibers.

The Grasp operation, presented in Fig. 1a, is a unit opera-tion that allows grasping of an individual fiber with themanipulators’ microgrippers. The Grasp operation requiresthe utilization of the Detect operation to determine the suitable

1 Common Object Request Broker Architecture (CORBA) is standarddefined by the Object Management Group that allows distributed com-ponents to communicate with each other.

J Micro-Bio Robot

Page 4: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

grasping points. After successfully locating the graspingpoints, microgrippers are moved to the found location (1).The microgrippers are closed after the desired location hasbeen reached (2). Multiple Grasp operations may take place atthe same time. This is the case when a fiber is lifted from thesubstrate in CFL.

The Pull unit operation2 is utilized to drag a fiber in a planewith a single microgripper. The direction of the Pull operationmay vary depending on the application, for example in thefiber-fiber bond breaking, the direction of the Pull operationshould be orthogonal to the fixed fiber. Another type ofapplication of the Pull operation is depicted in Fig. 1b, wherethe unit operation is utilized to measure friction between twofibers.

The Move operation is utilized to move a fiber from onelocation to another. It requires a successful execution of theGrasp operation, after which the fiber is lifted up from thesubstrate and moved into the desired location. The Moveoperation can be extended into a CoordinatedMove operation,where the fiber is grasped and moved with two microgrippers,as shown in Fig. 1c.

2.2 Characterization procedures

Different fiber characterization procedures can be created bycombining the described unit operations.

Fiber Pick and Place (FPP) combines unit operations De-tect, Grasp, Move, and Release. In the first phase, suitablegrasping points for a fiber are detected. Next, the Graspoperation is executed for the located grasping point. Thegrasped fiber is then moved to a desired location with theMove operation, where the fiber is placed on the substrate bythe Release operation.

Coordinated Fiber Lifting (CFL) sequence starts with theDetect operation that locates the coordinates of both fiber end-points. After successful detection, the Grasp operation isperformed for both end-points of the fiber simultaneously.

Finally, the fiber is lifted up from the substrate by performingthe Coordinated Move operation along z-axis. CFL can beextended to a fiber-fiber bond breaking sequence byperforming additional Detect, Grasp, Pull operations. First,the end of the free fiber must be identified with the Detectoperation, which is followed by the Grasp operation. Finally,the bond is destructed by performing the Pull operation on thefiber that is grasped only from one of its end points.

3 Experimental setup

The experimental microhandling platform, presented inFig. 2a, consists of three 4-DOF and one 3-DOF manipulatorsthat are assembled in a stacked gantry crane configuration.Grasping and moving of the manipulated fibers is performedwith the 4-DOF manipulators, denoted as Mani-1 (1), Mani-2(2) and Mani-3 (3). The 4-DOF manipulators are constructedfrom three linear micropositioners and a microgripper (SLC-1740, SG-1730-M, SmarAct GmbH). Amore detailed view ofMani-1 is provided in Fig. 2b. The used microgrippers, shownin Fig. 2c, have the thickness of 300 μm and the width of thejaw at the tip is 250 μm.

The 3-DOFmanipulator (4), here referred as Sample Stage,consists of two linear and one rotary micropositioners (SLC-1760, SR-4513-S, SmarAct GmbH). The samples are initiallyplaced on a circular glass slide that resides on top of SampleHolder (5). The positioning of the samples if performed withSample Stage’s linear positioners. The rotary axis providesmeans to align the fibers in parallel with the microgrippers, asdepicted in Fig. 2a.

The fibers and microgrippers are monitored with an imag-ing system that consists of two cameras, Manta GS-501,Allied Vision Technologies Inc.) denoted as Cam-1 andCam-2. Both of them are equipped with motorized optics(6–7, 12× Zoom, Navitar Inc.) and polarizers. The camerasare located above the microhandling setup. The angle betweenthe cameras is 30°. The illumination is designed so that it2 In Pull, fiber is pulled along its longitudinal axis to a desired location.

Fig. 1 a) Grasp: Gripper ismoved to desired location (1) andclosed (2) b) Frictionmeasurement as example of Pull;one of the fibers is pulled along itsx-axis c) Fiber transported toanother location with CoordinatedMove operation

J Micro-Bio Robot

Page 5: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

allows the identification of fibers, but also enables the trackingof the pose of the manipulators. Therefore, two differentapproaches are combined in the illumination. To acquirehigh-contrast images of fibers, a backlight, a polarizer andan analyzer were used in imaging. When there is a 90° phasedifference between the polarizer and the analyzer, thepolarizer blocks all the light that does not pass the fibers andthus changes its polarity. The polarizer and the backlightelements are located inside of Sample Holder so that it re-mains stationary while the samples are rotated during thealignment, as illustrated in Fig. 3. Manipulators are madevisible with the additional illumination element which simplyprovides ambient light. The illumination is presented in amore detailed manner in [32].

3.1 Control schemes

Communication between the PC unit and different low levelcontrollers is illustrated in Fig. 4. The low level control of themanipulators is performed with a manufacturer specific

Modular Control System (MCS). It consists of five ModularControl Units (MCU), each of which is responsible for con-trolling three micropositioners. MCUs 1–4 control the axis ofthe manipulators and MCU 5 is reserved for the control ofmicrogrippers. Each MCU implements a closed-loop positioncontrol individually for every controlled micropositioner.Communication between the MCS and the PC unit is serialcommunication over USB. The positioning and configurationcommands are issued from the PC unit through a manufactur-er specific API. WhenMCS receives a command, it internallyforwards it to the appropriate MCU, which then performs theclosed-loop positioning utilizing the micropositioner’s in-builtencoder as position feedback. The low level control of motor-ized optics is based on a similar control scheme asmanipulatorlow level control. The motors of Optics-1 and Optics-2 arecontrolled with Optics Control Units (OCU) OCU1 andOCU2, respectively. Internally, both OCUs implementclosed-loop control of zoom and focus motors. Feedback isprovided by the encoders that are integrated into the motors.Communication between OCUs and the PC unit is

Fig. 2 a) Overview of theexperimental setup: threemanipulators (1–3), Sample Stage(4), Sample Holder (5), andmotorized optics (6, 7); b) Close-up of manipulator Mani-1; c)Close-up of microgrippers jaws

Fig. 3 Sample Stage constructedfrom two linear and one rotarymicropositioners. Sample area isilluminated with a stationary ledarray through the polarizer

J Micro-Bio Robot

Page 6: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

implemented through serial communication ports. The cam-eras Cam-1 and Cam-2 communicate with the PC unit throughGigabit Ethernet connection. The PC unit is equipped with aseparate network adapter for each camera.

High level control is based on feed-forward control. Subtaskseither utilize image processing algorithms that calculate manip-ulator position set points from the obtained image data, ordirectly instruct themanipulator tomove to a predefined position.The high level control of Fiber Pick and Place (FPP) sequence isillustrated in Fig. 5. The figure neglects the low level controllersand describes only the connections between high level controllersand manipulators. FPP starts with subtask Approach. The sub-task utilizes high level image-based control algorithm (DetectGrasp Points), which obtains the grasping points of a selectedfiber. If suitable grasping points are successfully located, thecoordinates are transformed to the coordinate systems of Mani-1 and Mani-2. Approach subtask moves first Mani-1 to itsrespective grasp position (XM1, YM1, ZM1). After Mani-1 hasreached its position, Mani-2 is instructed to move to the graspposition (XM2, YM2, ZM2). Movement of both manipulators is a

sequence where they first move to the appropriate location in thexy-plane, which is followed by the z-directional approach to thefinal grasping position. After the manipulators have reached theirpositions, the task is continued by the execution of the Graspsubtask, where both of the grippers are closed simultaneously.This is followed by lifting the fiber up from the surface with theCoordinated Move subtask along the z-axis. After reaching thepredefined z-position, the grasped fiber is moved into desiredlocation in the Coordinated Move subtask. Finally, the fiber isreleased by the Release subtask. Subtasks Grasp, CoordinatedLift, CoordinatedMove, and Release utilize master/slave control,where Mani-1 acts as a master and Mani-2 is the slave.

3.2 Coordinate systems and actuator based calibration

The coordinate frames, illustrated in Fig. 6 are noted as {M1},{M2}, for Mani-1, and Mani-2 respectively. The manipulatorsare equipped with dot grids, where the dot diameter is 75 μm.Camera image coordinate frames {C1} and {C2} are mappedinto each manipulator’s coordinate system using actuator

Fig. 4 Low level control scheme of an individual MCS Control Unit

J Micro-Bio Robot

Page 7: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

based calibration. Relationship between image coordinates ofthe cameras and the coordinate systems of the manipulators isestablished by tracking of the location of the dot grids that areattached to the manipulators.

The calibration is performed by moving each of the ma-nipulators through a trajectory that contains 27 different posi-tions, where initial position is defined as Mnp0, where n de-scribes index of the manipulator. In each trajectory point, thelocation of the dot grid origin is detected in {C1} and {C2}from the images of Cam-1 and Cam-2, and mapped againstthe manipulator’s position in its own coordinate system. Therelation between camera image coordinates and metric coor-dinates can be described with a 3×4 camera matrix (P), whichfulfills the following equation:

x ¼ PX ð1Þ

where x is a homogeneous image coordinate point and X is ahomogeneous metric 3D coordinate point defined as

x ¼ u v 1½ �T ;X ¼ X Y Z 1½ �T ð2Þ

where u and v are the horizontal and vertical image coordi-nates and X, Y and Z are the 3D coordinates (1). When thecamera matrices have been defined for the both cameras, themetric 3D coordinates can be calculated from image point-image point correspondences between the images of the cam-eras by solving the equation pair produced from (1).

As a result, camera matrices MnC1P and Mn

C2P are obtainedindividually for each manipulator n. The origin of theestablished coordinate frame is located at the origin point ofthe manipulator’s dot grid at Mnp0. The coordinates are furthertransferred to the coordinate system of the end-effector byadding the offset between the end-effector and the dot grid tothe coordinates provided. An example image obtained fromCam-1 during the calibration is presented in Fig. 7. Thecalibration sequence utilizes the camera calibration functionsof OpenCV image processing library.3 The principal of thecalibration is presented in more details in [33].

3.3 Grasp point detection algorithm

Grasp Point Detection Algorithm (GPDA), presented inFig. 8, enables the location of suitable grasp points. The fiber3D profile is approximated from an image pair provided bythe cameras, and the grasp points are detected from the profileby applying certain selection criteria.

The procedure to obtain grasp points is as follows. First, thebinary images of fibers are formed from the camera image pair(a, c) by global threshold operation. As described in Section 3,the contrast between the dark background and the bright fibersis sufficiently high and thus simple threshold operation can beused in the segmentation. The centerline of the fiber (b, d) isestimated by applying morphological thinning to the binaryimages representing the fiber. In the next phase, the corre-spondence points between the centerline images are estimatedby calculating the curvature of the fibers, and linking thepoints with high curvature with each other. The segmentsbetween the correspondence points are also linked with eachother. Finally, the 3D profile in the coordinate system of themanipulator n can be calculated by solving the equation pair:

C1x ¼ C1MnP

MnX

C2x ¼ C2MnP

MnX

(ð3Þ

where MnC1xi and Mn

C2xi are the corresponding image coordinates,and MnXi is the 3D coordinate in the coordinate system ofmanipulator n. The final result of located grasp points isshown in Fig. 8e.

Grasp points are searched from the fiber profile iterativelyby applying following user defined criteria:

3 www.opencv.org

Fig. 5 Control scheme of Fiber Pick and Place (FPP) task

J Micro-Bio Robot

Page 8: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

1. Minimum and maximum length for the segment from thegrasp point to the nearest fiber end

2. Maximum angle for the segment grasped with the gripperjaws

3. Minimum length for the segment between the graspingpoints

4. Maximum difference in the z direction

If multiple grasp points fulfilling the criteria are found, theones having the longest segment between are returned. Also,the image coordinates of the grasp points are calculated byapplying (1) and returned for visual assessments. Figure 8presents the phases of the fiber grasp point detection algo-rithm. The parameters for the four criteria presented abovewere 100–2,000 μm, 15°, 1,000 μm, 100 μm, respectively.

4 Control software

The developed control software utilizes Rock robotic softwareframework, which is a close relative of Orocos robotic soft-ware framework. Rock relies on the component model ofOrocos, but takes an alternative approach for system deploy-ment and provides wider variety of off-the-shelf components.In comparison with Orocos, Rock provides extended systemlevel management capabilities. The aspects supporting theselection of Rock were (i) well defined component model,(ii) support for real time operating systems4 (Xenomai, RTAI),(iii) active English-speaking user group, (iv) good documen-tation, (v) ability to integrate with other RFSs (vi) wellestablished mechanism for component sharing and version

control. The motivation for requiring a strict component mod-el is that it streamlines the component development.

Other investigated potential options were Orocos Real-Time Toolkit (Orocos RTT), OpenRTM-aist, OPRoS andROS. As mentioned earlier, Orocos RTT is closely related toRock. In this work, Rock was preferred over Orocos RTTmainly due to the utilization of more commonly used man-agement level scripting language (Ruby). OpenRTM-aist is anRSF developed by the Japanese National Institute of IndustrialScience and Technology (AIST). Software components devel-oped with OpenRTM-aist follow Robotic Technology Com-ponent (RTC) specification from Object Management Group(OMG). It supports multiple operating systems (Linux, OS X,Windows) and programming languages (C++, Java, Python).OPRoS is a Korean effort for developing component-based

4 This work relies on standard Linux kernel, but the real-time support isrequired in the future development.

Fig. 7 Calibration dot grid attached to gripper jaw. The dots of the gridare 75 μm in diameter

Fig. 6 Coordinate frames ofMani-1 {M1} and Mani-2 {M2},and locations of dot grids

J Micro-Bio Robot

Page 9: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

robotic software. OPRoS does not follow RTC specification,but defines its own component model that is similar to RTC.OPRoS components are written in C++. Both OpenRTM-aistand OPRoS provide Eclipse-based graphical user interface forconstructing and connecting the robotic software components.

Rock was preferred over OpenRTM-aist, because Rockhides the middleware implementation from the componentdeveloper. Middleware specific knowledge is only requiredwhen new complex data types have to be created. Further-more, Rock has a very active English-speaking communitywhich has proven to be the best source of information in manyimplementation related issues. OPRoS does not currentlyprovide real-time capabilities. However, there are ongoingefforts in this field [34]. Further, OPRoS did not provideintegration with ROS at the time of deciding upon the utilized

RSF. The rationale of requiring integration with ROS is that itis a widely used RSF and most of other RSFs and roboticsoftware tools integrate with ROS. The authors do not want tolimit themselves to one single RSF and the ability to integratewith ROS provides certain flexibility, as different systems orsubsystems can be built on different RSFs and intercommu-nication can be implemented with ROS.

4.1 Orocos component model

Orocos component model defines two distinctive concepts forcommunication: services and flow ports. Service interfacesare access points through which the component provides orrequires operations. Service operations are used in a similarmanner as function calls are utilized in programming

Fig. 8 Implementation of GraspPoint Detection Algorithm(GPDA). The algorithm’s inputimages (a, c) are processed bysubsequent threshold andmorphological thinningoperations (b, d). Fiber grasppoints (e) are obtained byapplying user defined criteria

J Micro-Bio Robot

Page 10: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

languages. Flow ports implement a publish-subscribe scheme,where a component with an output port acts as a data publisherand components with input ports can subscribe to listen to theincoming data. The life cycle of a component is managed witha simple state-machine each component implements. Uponinstantiation, Orocos components enter PreOperational state,where the component remains until it is configured. After asuccessful configuration, a transition to Stopped state is per-formed. When started, the component enters Running state,where it remains until it is stopped and transitions to Stoppedstate. The final state in component’s lifecycle is Cleanup afterwhich the component is destructed. Orocos component modelalso includes recoverable and unrecoverable error states,which allow the developers to define appropriate actions incase of exceptions.

The design of the components and system level composi-tion in this work is guided by two design patterns introducedby Rock. On the component level, Rock proposes decouplingof the component’s functionality from the component itself.That is, the functionality should be implemented as a separateshared object, whose behavior is then determined at the com-ponent level. The second design pattern discusses communi-cation between component instances and management-component communication. Components should also interactwith other components through flow port, i.e. componentservice interfaces are considered as access points for systemmanagement. It is worth noting that such design patterns aremere recommendations, but following those increases cohe-sion across the developed system.

4.2 Software components

In this work, we combined in-house developed componentswith Rock off-the-shelf components in order to implement theunit operations. The utilized components can be categorizedinto two different levels: low level hardware interface compo-nents and high level components that implement supportivefunctionality, algorithms, and user interface elements that arerequired in the fiber level characterization procedures. Themanipulation tasks are implemented as scripts that orchestratethe run-time behavior of different component instances. Theused off-the-shelf components are an image acquisition com-ponent (CameraProsilica), and a gamepad control component(JoystickTask) that is used for teleoperation. The in-housedeveloped components include two hardware interface com-ponents (MCSController and NavitarController), two algo-r i t hm i c c omponen t s (De t e c tG r a spPo i n t s andCoordTransform), a gamepad management component(GamepadMgr), and an image stream aligner component(ImageAligner).

In the current phase, our component library includes twohardware interface components (Fig. 9), which enable com-munication with the hardware of the microhandling platform.MCSController component provides the means to communi-cate with SmarAct’s MCS micropositioner controller. Inter-nally, MCSController implements a manipulator abstractionthat allows the user to define which micropositioners form amanipulator. The defined manipulators are controlled eitherthrough the component’s service interface or through inputflow port (mani_pos_in). The first is suitable for request-response style cases, such as the execution of different se-quences. The latter can be used for the implementation ofcontrol loops (e.g. visual servo applications), where severalcomponents are chained with each other by connected dataflow ports.MCSController has one static and multiple dynam-ic output flow ports. For each defined manipulator, the com-ponent implements an additional output port that providesinformation about the manipulator’s status and position(mani<n>_state_out, where n represents the index of themanipulator). MCSController also includes master/slave con-trol option at manipulator level.

Master/slave control is implemented by utilizing atriggering mechanism of the low level controller, whichallows the submission of different instruction for eachMCU followed by a simultaneous triggering of thesubmitted instructions. Behavior of MCSController isevent-based; when all manipulators are inactive (i.e.not moving) the component is in IDLE state. The com-ponent is woken only when either a function of theservice interface is called or new data arrives at theinput port. In this case, a state transition to EXEC takesplace. The component remains in EXEC state until allmanipulators have been stopped. The manipulator spe-cific output ports submit data only when the componentis active and the manipulator respective to the particularport is moving. The static port (status_out) sends outinformation regarding the state changes of thecomponent.

NavitarController component provides asynchronous serialcommunication with Navitar optics controllers. Similar toMCSController, it provides a service interface and an alterna-tive flow port input (optics_pos_in), which can be utilized toimplement closed-loop control schemes. The state and posi-tion of the controlled motorized optics is published throughthe components output (optics_state_out). Behavior ofNavitarController is similar to that of MCSController. Thecomponent executes commands when new data is receivedthrough input port optics_pos_in or one of the interface func-tions is called.

J Micro-Bio Robot

Page 11: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

Image acquisition is based on Rock’s CameraProsilicacomponent, which provides the ability to control and acquireimages from GigE Vision cameras that are manufactured byAllied Vision Technologies Inc. Camera configuration is im-plemented as properties of the Orocos component model andimage data is accessible through subscription to the compo-nent’s output flow port (frame_out). The run-time behavior ofCameraProsilica is asynchronous. It waits for images fromthe connected camera and publishes them through the outputflow port. Hence, the execution of the component is deter-mined by the underlying manufacturer specific library.

Another off-the-shelf component called JoystickTask pro-vides the ability to communicate with gamepads, which areused for teleoperated control of the microrobotic platform.The component provides different output flow ports, such asfor different types of control applications. In this case, the rawcommand structures published through specific output flowport (raw_command) are utilized. Unlike the other low levelcomponents, JoystickTask has periodic behavior. The compo-nent polls the outputs of a gamepad with a given samplingfrequency and publishes the results through the output flowports.

The high-level components consist of a set of user interfacecomponents, two algorithmic components, a gamepad man-agement component, and an image stream aligner component.Algorithmic component DetectGraspPoints implements thegrasp point detection algorithm (GPDA) described in Sec-tion 3.3. DetectGraspPoints is accessible only through itsservice interface and hence is designed to be accessible onlyfrom the management layer. The second algorithmic compo-nent CoordTransform performs coordinate system transfor-mations from image point coordinates to manipulator coordi-nate systems. The camera matrices obtained through the cal-ibration process described in Section 3.2 are loaded intoCoordTransform instance, where they are identified with therespective name of the manipulator.CoordTransform provides

a service interface, but it also includes a data flow port alter-native. ImageAligner is an event-based component that trig-gers to its input ports frame1_in and frame2_in. ImageAlignerpublishes image pair through its output flow port (frames_out)when it has received images to both of its input ports. Thecomponent can be configured so that it publishes images onlywhen the time stamps of the images are within a given timeinterval. GamepadMgr allows the users to control differentmanipulators and motorized optics with a gamepad. The com-ponent implements a state-machine, where a change of stateswitches the incoming data flow either to one of the motorizedoptics instances (optics1, or optics2) or to theMCSControllerinstance (mcsController). Fig. 10 presents the runtime com-position of the component instances.

5 Experiments and results

Three experiments were performed to evaluate the perfor-mance of the developed Fiber Pick and Place (FPP) andCoordinated Fiber Lift (CFL) sequences. The first experiment(Experiment 1) concentrated on Grasp Point Detection Algo-rithm (GPDA). The purpose of this experiment was to evalu-ate the performance of the algorithm prior to manipulationexperiments. In Experiment 2, the focus was set on the eval-uation of the repeatability of Coordinated Fiber Lift (CFL)sequence. Experiment evaluated the overall performance ofthe system by executing Fiber Pick and Place (FPP) sequence.The experiment concentrated on the success rate and through-put of consecutive unit operations Grasp, Move, and Releaseby placing fibers on adhesive substrate, which simulatedsample preparation for SEM.

Prior to the experiments, manipulators Mani-1 and Mani-2were calibrated with the calibration procedure described inSection 3.2. The reprojection errors were evaluated to be 0.54

Fig. 9 Developed hardwareinterface componentsMCSController,NavitarController and Rock off-the-shelf componentsCameraProsilica andJoystickTask

J Micro-Bio Robot

Page 12: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

and 0.84μm, respectively for Cam-1 and Cam-2. The samplesused in the experiments, softwood pulp (kraft pulp, finesremoved) were prepared with the following procedure. Firstlow concentrate solution containing the fibers was pipetted onTeflon plates. After which, the samples were dried in an ovenfor 45 min in temperature of 70 C. Finally, the fibers weremanually placed on circular glass slides that can be fitted ontop of Sample Holder. Each slide contained 5–10 fibers.

In Experiment 1, the performance of the fiber grasp pointdetection algorithm was evaluated with 40 samples. The glassslide containing the samples was placed on top of SampleHolder and the selected sample was positioned under the fieldof view of both cameras in a teleoperated mode. In the nextphase, the operator selected appropriate Region-of-Interest

(ROI) that contained a single fiber, and adjusted the imagethreshold levels accordingly through a graphical user inter-face. In the final step, GPDAwas executed. The success of thealgorithm was evaluated by reprojection of the found graspingpoints into the image coordinates. Reprojected grasping pointsare illustrated in Fig. 11, where the located grasping points arepresented as red and yellow circles, respectively for Mani-1and Mani-2.

GPDAwas able to locate suitable grasping points from 35out of 40 samples, yielding a success rate of 87.5 %. Execu-tion time of the algorithm was in average 1.74 s with astandard deviation of 0.29 s. The worst case execution timewas 2.31 s. In two of the failed cases, the failure was due to thefiber geometry; the given boundary conditions for suitable

Fig. 10 Run-time composition of low-level components. Management layer can subscribe to any component instances output ports, publish into theirinput ports and call the service interfaces of the component instances

J Micro-Bio Robot

Page 13: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

grasping points could not be fulfilled. In the threeremaining cases, presented in Fig. 12, the reconstructionof the fiber from the 2-d images failed. The left-handside of each image is acquired from Cam-1 and theright-hand side images are from Cam-2. In Fig. 12a,the fiber reconstruction failed due to inconsistent image

information at the defocused fiber ends. In Fig. 12b, thefiber is broken, i.e. parts of the fibers are invisible asthere is not a contrast difference between the indicatedparts of the fiber and the background. In Fig. 12c, thelower end of the fiber includes a hook shape which isnot sufficiently visible on the left-hand side (Cam-2)

Fig. 11 Examples of successful fiber grasping point locations for different fibers. Located grasping points are presented as red and yellow circles,respectively for Mani-1 and Mani-2. The left-hand side of each image pair is acquired with Cam-1 and right-hand side images are from Cam-2

J Micro-Bio Robot

Page 14: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

image. The inconsistent information between the cameraprojections causes the fiber reconstruction phase ofGPDA to fail.

The fibers utilized in Experiment 1 are representative sam-ples of the fibers that were also used in other the experiments.Their curl, length and defibrillation level have high variances.The length distribution of the fibers used in Experiment 1 isshown in Fig. 13.

Experiment 2 focused on the evaluation of the developedCFL sequence by automatically picking individual fibers fromthe glass slide positioned on top of Sample Holder. UsingMani-1 and Mani-2 and lifting them up. The experimentconsisted of a repetition of CFL sequence on 40 samples.For each sample, the total duration was measured. The successwas evaluated by the operator, by visually inspecting thesuccess of the Grasp operation and the entire sequence. Thesuccess of the grasp point detection algorithm was evaluatedin the same manner as in Experiment 1. The performance ofthe grasp point detection algorithm was on a consistent level

with Experiment 1. CFL sequence was performed successful-ly in 33 out of 40 cases, which results in a success rate of82.5 %. The average execution time of CFL sequence was51.45 s with a standard deviation of 19.5 s, the distribution ofexecution times is presented in Fig. 14 as a histogram. Themost common reason for failure was that the grasp pointdetection algorithm was unable to find suitable graspingpoints (4 failures). The other unsuccessful cases failed duringthe Grasp operation. In two cases, the grippers of Mani-1 andMani-2 did not close simultaneously, which causeddispositioning of the fiber during the Grasp operation. In onecase, the fiber moved out of the field of view while thegrippers were approaching it. This behavior was most likelydue to the electrostatic charge of the fiber. In one case thegripper of Mani-1 did not grasp the fiber correctly and theother end of the fiber was released during CFL sequence.

Experiment 3 aimed to test the performance of the FiberPick and Place (FPP) sequence. The performed sequence wasidentical with CFL sequence, but it is extended with an

Fig. 12 Failed test cases of Experiment 1. Failures were caused by defocused parts of the samples (a), irregular behavior of fibers under polarized light(b), and geometries that were not visible in both images (c)

1 1

2

0

1

0

4

2

3

4 4

0 0 0

4

3

1

3

0

3

1

0

1

2

3

4

1600

-170

0<1

800

<190

0<2

000

<210

0<2

200

<230

0<2

400

<250

0<2

600

<270

0<2

800

2900

<300

0<3

100

<320

0<3

300

<340

0<3

500

<360

0<3

700

Occ

uren

ces

Length [µm]

Fig. 13 Length distribution ofthe fiber lengths in micrometers.The results are obtained fromExperiment 1

J Micro-Bio Robot

Page 15: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

additional Coordinated Move and Release operations. In thelatter, the grasped fibers are placed on a predefined location onan adhesive substrate. The goal was to place the fibers in nextto each other so that the gap between two fibers is 0.5 mm.FPP sequence was executed 30 times, where 73 % of theexperiments were successful. The average execution time ofFPP was 59.2 s with a standard deviation of 8 s. The main

reason for failure was that the placed fibers did not adhere tothe surface (5 cases). The other failures were due to theunsuccessful execution of GPDA (3 cases). In all 27 cases,where GPDA was successfully executed, the Grasp and theCoordinate Move operations were also successful. Experi-ment 3 did not focus on the positioning accuracy of the placedfibers. However, as positioning is an important aspect in manyreal world applications of FPP, few remarks are discussed inthe following. Figure 15 presents a typical result of consecu-tive FPP sequences. The distances between the fibers, alongthe x-axis, varied in all the experiments between 0.45 and0.8 mm. One of the reasons for large variation is the grasppoint selection criteria. It causes the fibers not to be graspedfrom their end-points, but from points that fulfill certainboundary conditions as discussed in Section 3.2. Further, thecurrent FPP does not contain a step that would straighten thefiber prior to the Release operation. Positioning accuracy ofthe fiber in FPP can be increased by developing more suitablecriteria for GPDA, and by an additional unit operation thatstraightens the fiber. Another source of error that could beimproved was the opening gap of the grippers during theRelease operation. In these experiments, identical gripperopening was utilized for all fibers. In several cases, the gripperjaw pushed previously placed fibers while the gripper wasopened in the Release operation. This issue can be addressedby implementing a visual feed back into the release phase.

In addition to the automation tests, a teleoperated experi-ment was carried out. It consisted of fifteen subsequent Coor-dinated Fiber Lift sequences that were performed by an expe-rienced user. The success rate of the reference experiment wasidentical to that of automated CFL; 12 out of 15 experiments(80 %) resulted in successful grasping and lifting of themanipulated fiber. The average time spent on teleoperated

12

5

12

7

3

1 1 10 0

10

0

2

4

6

8

10

12

14

<30 <40 <50 <60 <70 <80 <90 <100 <110 <120 <130 <140 More

OCC

URE

NCE

STime (seconds)

Fig. 14 Distribution ofCoordinated Fiber Lift (CFL)execution times. The averageexecution time was 51.45 s

Fig. 15 Four fibers placed on an adhesive substrate with Fiber Pick andPlace (FPP) sequence

J Micro-Bio Robot

Page 16: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

CFL was 131 s with a standard deviation of 30 s, which issignificantly longer than execution of automated CFL. Distri-bution of teleoperated CFL is show in Fig. 16. Automation ofCFL provides a significant performance increment in terms ofexecution time. The results of all experiments are recapitulat-ed in Table 1.

6 Conclusions and future work

In this paper, we have demonstrated automated unit operationsthat are an essential part of the effort to develop a fullyautomated microrobotic platform for the characterization ofnatural fibrous materials. Automated Fiber Pick and Place,and Coordinated Fiber Lifting procedures were successfullydemonstrated. The automated Coordinated Fiber Lifting se-quence had a success rate of 82.5 % and its execution took51.5 s on average. The teleoperated reference experiment hasthe same success rate, but the execution time was significantlylonger being 131 s on average. The automated sequence takesless than 40 % of the time required in the teleoperated mode.The developed Fiber Pick and Place sequence was success-fully used for placing samples on adhesive substrate. Howev-er, it suffers from inadequate positioning accuracy to be fully

utilized. The positioning accuracy of Fiber Pick and Place willbe enhanced in the future, by adjusting Grasp Point DetectionAlgorithm specifically for this operation, and by the develop-ment of an additional unit operation for straightening thegrasped fiber. Moreover, a high-level visual servo algorithmfor the positioning of the fibers is required in order to auto-matically create man-made fiber-fiber bonds.

The presentedwork reported, for the first time, the utilizationof a mature robotic software framework and component-modelas a basis for microrobotic control software. The presentedexperiments demonstrate that existing robotic software frame-works provide an efficient alternative for in-house developedsoftware platforms that are commonly used by the microroboticcommunity. The approach has proven to increase productivity,as software components have significantly increased the reuseof the developed source code. Moreover, the approach enablesan efficient way for sharing software modules within the com-munity by encapsulating the functionality into components.The next steps towards full automation will include the imple-mentation of different algorithms, such as the detection of fibersthat cannot be grasped and visual servo controllers for fiber andfiber-bond manipulation tasks. Further, we aim to developsoftware design patterns that specifically target at softwaredevelopment in micro- and nanorobotics domain.

0

1

2

3

80 90 100 110 120 130 140 150 160 170 180 190 200

Occ

uren

ces

Time [s]

Fig. 16 Distribution of averagetime spent on teleoperatedCoordinated Fiber Lift sequence

Table 1 Summary of experi-ments 1–3 and manual coordinat-ed lift

Experiment Number ofsamples

Average executiontime [s]

Successrate [%]

Experiment 1: Grasp Point Detection Algorithm (GPDA) 40 1.74 87.5

Experiment 2: Coordinated Fiber Lift (CFL) 40 51.5 82.5

Experiment 3: Fiber Pick and Place (FPP) 40 59.2 73

Manual CFL 15 131 80

J Micro-Bio Robot

Page 17: Robotic software frameworks and software component models in the development of automated handling of individual natural fibers

References

1. Ververis C, Georghiou K, Christodoulakis N, Santas P, Santas R(2004) Fiber dimensions, lignin and cellulose content of various plantmaterials and their suitability for paper production. J Ind Crops andProd 19(3):245–254

2. Becattini G, Mattos LS, Caldwell DG (2014) A fully automatedsystem for adherent cells microinjection. IEEE J Biomed andHealth Inform 18(1):83–93

3. McDaid AJ, Haemmerle E, Xie SQ, Aw KC (2013) Design, analysis,and control of a novel safe cell micromanipulation systemwith IPMCactuators. J Mech Des 135(6):10, 061003

4. Park J, Jung S-H, Kim Y-H, Kim B, Lee S-K, Ju B, Lee K-I (2004)An integrated bio cell processor for single embryo cell manipulation.In: IEEE/RSJ International Conference on Intelligent Robots andSystems. pp. 242–247

5. Kallio P, Kuncova-Kallio J (2006) Capillary pressure microinjectionof l iving adherent cells : challenges in automation. JMicromechatronics 3(3–4):3–4

6. Georgiev A, Allen PK, Edstrom W (2004) Visually-guided proteincrystal manipulation using micromachined silicon tools. In:IEEE/RSJ International Conference on Intelligent Robots andSystems. pp. 236–241

7. Arai F, Sakami T, Maruyama H, Ichikawa A, Fukuda T (2002)Minimally invasive micromanipulation of microbe by laser trappedmicro tools. In: IEEE International Conference on Robotics andAutomation. pp. 1937–1942

8. Inoue K, Arai T, Tanikawa T, Ohba K (2005) Dexterous microma-nipulation supporting cell and tissue engineering. In: IEEEInternational Symposium on Micro-NanoMechatronics and HumanScience. pp. 197–202

9. Stolle C, Bartenwerfer M, Celle C, Simonato J, Fatikow S (2013)Nanorobotic strategies for handling and characterization of metal-assisted etched silicon nanowires. IEEE/ASME Trans onMechatronics 18(3):887–894

10. Ye X, Zhang Y, Ru C, Luo J, Xie S, Sun Y (2013) Automated pick-place of silicon nanowires. IEEE Trans on Autom Sci and Eng 10(3):554–561

11. Tamadazte B, Le Fort-Piat N, Dembélé S, Fortier G (2011) Roboticmicromanipulation for microassembly using monoview andmultiscale visual servoing. Mechatronics, IEEE/ASME Trans 16(2):277–287

12. Xie H, Régnier S (2009) Three-dimensional automated micromanip-ulation using a nanotip gripper with multi-feedback. J MicromechMicroeng 19(7):9, 075009

13. Probst M, Hürzeler C, Borer R, Nelson BJ (2009) A microassemblysystem for the flexible assembly of hybrid roboticMEMS devices. IntJ Optomechatronics 3(2):69–90

14. Cecil J, Jones J (2014) VREM: An advanced virtual environment formicro assembly. Int J Adv Manuf Technol 72(1–4):47–56

15. Saketi P, Kallio P (2011) Microrobotic platform for making, manip-ulating and breaking individual paper fiber bonds. In: IEEEInternational Symposium on Assembly and Manufacturing. pp. 1–6

16. Saketi P, von Essen M, Mikczinski M, Heinemann S, Fatikow S,Kallio P (2012) A flexible microrobotic platform for handling micro-scale specimens of fibrous materials for microscopic studies. JMicrosc 248(2):163–171

17. Saketi P, Kallio P (2011) Measuring bond strengths of individualpaper fibers using microrobotics. In: Progress in paper physics sem-inar. pp. 199–203

18. Essen M von, Hirvonen J, Saketi P, Kallio P (2011) Automatedgrasping in manipulation of individual paper fibers. In:Manipulation, Manufacturing and Measurement on the Nanoscale.

19. Mikczinski M, Josefsson G, Chinga-Carrasco G, Gamstedt EK,Fatikow S (2014) Nanorobotic testing to assess the stiffness proper-ties of nanopaper. Robot, IEEE Trans on 30(1):115–119

20. Mikczinski M, Nguyen HX, Fatikow S (2013) Assessing TransverseFibre Properties Compression and Artificial Hornification byPeriodic Compression. In: 15th Fundamental Research Symposium,the Fundamental Pulp and Paper Research Society.

21. Saketi P, Mikczinski M, Fatikow S, Kallio S (2013) Method forinvestigation of aged fibre-fibre bonds with micro and nanorobotictools. In: 15th Fundamental Research Symposium, the FundamentalPulp and Paper Research Society.

22. Gerkey BP, Vaughan RT, Stoy K, Howard A, Sukhatme GS, MataricMJ (2001)Most valuable player: A robot device server for distributedcontrol. In: IEEE/RSJ International Conference on Intelligent Robotsand Systems. pp. 1226–1231

23. Brooks A, Kaupp T, Makarenko A, Williams S, Orebäck A (2007)Orca: a component model and repository. In: Software engineeringfor experimental robotics. Springer, pp 231–251

24. Soetens P, Bruyninckx H (2005) Realtime hybrid task-based controlfor robots and machine tools. In: IEEE International Conference onRobotics and Automation. pp. 259–264

25. Quigley M, Conley K, Gerkey B, Faust J, Foote T, Leibs J, WheelerR, Ng AY (2009) ROS: an open-source Robot Operating System. In:ICRAworkshop on open source software, vol 3.2.

26. Metta G, Fitzpatrick P, Natale L (2006) YARP: yet another robotplatform. Int J Adv Robot Syst 3(1):43–48

27. Ando N, Suehiro T, Kotoku T (2008) A software platform forcomponent based rt-system development: Openrtm-aist. In:Simulation, Modeling, and Programming for Autonomous Robots.Springer, pp 87–98

28. Song B, Jung S, Jang C, Kim S (2008) An introduction torobot component model for OPRoS (Open Platform forRobotic Services). In: Workshop Proceedings of Intl. Conf.on Simulation, Modeling and Programming for AutonomousRobots. pp. 592–603

29. Calisi D, Censi A, Iocchi L, Nardi D (2008) OpenRDK: a modularframework for robotic software development. In: IEEE/RSJInternational Conference on Intelligent Robots and Systems. pp.1872–1877

30. Schlegel C, Worz R (1999) The software frameworkSMARTSOFT for implementing sensorimotor systems. In:IEEE/RSJ International Conference on Intelligent Robots andSystems. pp. 1610–1616

31. Joyeux S, Albiez J (2011) Robot development: from components tosystems. In: 6th National Conference on Control Architectures forRobots.

32. Hirvonen J, HänninenA, Kallio P (2014)Design and Implementationof an Illumination System for Microrobotic Paper Fiber Studies. In:IEEE International Conference on Robotics and Automation.(accepted)

33. Hirvonen J, Kallio P (2013) Scale and Rotation InvariantMicrogripper Detection that Uses a Planar Pattern inMicrorobotic Two-View System. In: IFAC Symposium onMechatronic systems.

34. Jang C, Song B, Jung S, Lee K-H, Kim S (2011) Real-timesupporting of OPRoS component Platform. In: 8th InternationalConference on Ubiquitous Robots and Ambient Intelligence. pp.640–641

J Micro-Bio Robot