FibreMap FP7 608768

1

FibreMap 608768

Automatic Mapping of Fibre Orientation for Draping of Carbon Fibre Parts

SEVENTH FRAMEWORK PROGRAMME NMP Priority

Deliverable D3.4: “Integrated robotic system for 3D part scanning”

Due date of deliverable: 2016-05-31 Actual submission date: 2016-06-15

Start date of project: 1st September 2013 Duration: 36 months Coordinator: Christian Eitzinger Profactor Revision 1 Lead Beneficiary: IT+Robotics Contributions by: -

Project co-funded by the European Commission within the 7th Framework Programme

Dissemination Level PU Public X PP Restricted to other programme participants (including the Commission

Services)

RE Restricted to a group specified by the consortium (including the Commission Services)

CO Confidential, only for members of the consortium (including the Commission Services)

FibreMap FP7 608768

2

Contents

Contents .......................................................................................................................... 2

1. Structure and Purpose of this Document .................................................................. 3

2. The inspection robotic system .................................................................................. 3

3. Workcell simulation .................................................................................................. 5

4. Coverage and path planning overview ..................................................................... 7

5. Robotic system results ........................................................................................... 10

6. Further information ................................................................................................. 10

FibreMap FP7 608768

3

1. Structure and Purpose of this Document

The purpose of this document, classified as “prototype deliverable”, is to describe the robotic system developed in the project and used for automated scanning of fibre orientation on carbon fibre parts. It includes the design and development of a robotics work cell, described in section 2. The work cell allows performing a full scan given a complex product (almost arbitrary shape) as input and provides as output the 3D model (point cloud) describing fiber directions (vector per point). The inspection task is supported by a software architecture composed of two modules. The first module is an offline programming software (OLP) provided by IT+Robotics, it is in charge of the process simulation and path planning in order to automatically generate the robot’s path for the inspection. The software needs three inputs: the product, the work cell description and the image acquisition model of the inspection sensor (in this project, Profactor’s “FScan” camera). Given this information, the software basically solves the “coverage planning” problem that ensures that the whole product is fully scanned. This software has kinematic and dynamic models of the robot to be used for the task in order to obtain a simulation to avoid collisions. The output of this first module is a robot inspection trajectory for the product. This module will be described in sections 3 and 4. The product inspection is performed while the robot is moving along the generated trajectory.

The inspection is in charge of the other module, an image acquisition software developed by project partner UNIPD. The software collects multiple and overlapping images (using the “FScan” sensor) while the robot is moving. Scanning of 3D parts and fibre mapping require precise information about the position and orientation of the camera by the time at which each single image is taken. For this reason, a precise synchronization of the robot motion and camera acquisition is required. The acquired dataset is processed to obtain an accurate fibre angle measurement. The robotic system has been deployed, tested and upgraded since the start of the project. The project lasted for three years and the robotic system results will be presented in section 5.

2. The inspection robotic system

The FibreMap robotic system is composed of a robot equipped with a particular sensor which allows to measure fibre orientation, as shown in Figure 2a. The work cell set up at the University of Padova is shown in Figure 2b. The scanning robot is surrounded by fences with proper safety measures allowing to immediately stop the robot if someone enters the robot’s operative area.

Figure 1. Software used for testing the generation of the inspection path.

FibreMap FP7 608768

4

(a) Robot scanning a test part (b) Work cell at UNIPD

Figure 2. Pictures of the work cell set up at UNIPD. The robot used is a Comau Smart5-SiX, an anthropomorphous robot with six axes / degrees of freedom and a payload of 6kg at the wrist. Its repeatability is +/- 0.05 mm. The robot is equipped with a Comau C4G control unit which maintains a real-time communication with the robot every 2 ms and which is endowed with the “Open” interface, which allows the control by using an external PC. As depicted in Figure 3, the C4G module is connected to a PC running Ubuntu 13.04 and the RealTime Application Interface (RTAI) for Linux, which allows writing applications while preserving real-time communication with the robot controller. This PC is used to listen to TCP position messages sent by the path planning module and sends them to the robot controller. If more messages are received, they are stored in a First-In-First-Out queue, which is then read by the robot control thread.

The real-time PC is also responsible for the streaming of the robot positions and time stamps over the network by means of UDP messages, so that every PC connected to the network can know the position of the robot every 2 ms. Another PC, running Windows 7, is connected by Ethernet connection to the Linux PC. This PC is responsible for all the high-level tasks like:

• the generation of the scanning path; • the synchronization of robot and camera time stamps; • the image acquisition; • the fibre angle computation; • the projection of the fibre angles to the 3D model of the part.

Figure 3. Different tasks are distributed to the different PCs.

FibreMap FP7 608768

5

The core components of the fibre angle sensor (“FScan” sensor) are a light ring with 96 LEDs and a PhotonFocus MV-1-D1312 GigE Camera. LEDs can be switched on and off pair-wise. The resolution of the camera is 1312x1024 pixels. The lens attached to the camera is a SKR STD XENON 17/0,95. A dedicated microcontroller triggers the camera for image capturing and at the same time switches on and off groups of neighboring LEDs. The sensor is powered over two pins of a D-Sub (DB-25) jack. A serial RS485 connection also runs over this D-Sub jack and connects the controlling PC with the microcontroller. Image data is delivered by the sensor over Gigabit Ethernet via a standard RJ45 jack. An upgraded version of the sensor (“FScan-H2”) includes 288 Luxeon Z ES SMD LEDs that can be switched on and off pair-wise. Compared to the first version of the sensor the light emitting area of LEDs was reduced from 5mm to 1.5mm. A more well-defined light circle is the result. The new sensor also includes an Ethernet switch which enables communication through Gigabit Ethernet with camera and light controlling electronics within the sensor. In contrary to the old sensor, no separate RS485 connection is therefore required.

3. Workcell simulation

The whole work cell developed at UNIPD can be simulated thanks to the offline programming software developed by IT+Robotics. This allows to automatize the definition of the fibre parts inspection cycle.

The work cell and process simulation is organized in different stages. In the first stage, the virtual work cell has to be defined. To this end, the work cell components have been

Figure 4. WCS Robot Plugin

FibreMap FP7 608768

6

designed with the Component Editor (CE) application. Every component is represented as a kinematic chain that is a set of links connected to each other by joints to form a tree structure (or more generally a graph). Every link has one or more degrees of freedom relative to its parent link. In particular, in the Fibremap work cell, two types of components are needed: the inspection device and the robot manipulator. In order to help the user, the CE has been customized with two plugins:

• A robot plugin (see Figure 4) to define the Denavit-Hartenberg robot parameters, set up axes limits, speeds and custom inverse kinematics. Using this plugin a virtual Comau Smart5 SiX has been created.

• A camera plugin that is used for handling the simulation of the second important component, the sensor vision system. For the demonstrator a virtual model of Profactor’s “FScan” sensor has been designed.

Each component can be enhanced with custom reference frames that are useful for getting and setting particular transformations between the robot and the component. For instance, a particular reference frame has been defined at the optical centre of the camera. This information is necessary for two main reasons:

• To place the virtual camera correctly with respect to the robot • In the coverage planner algorithm, to specify the position and the distance of the

camera with respect to the surface of the product. The virtual components have been used in WorkCell Editor (WCE) application to set up the corresponding virtual work cell of the UNIPD robotic system. The position of these components with respect to the robot is crucial for a meaningful inspection. Therefore, the application has been enhanced with the WorkCell Editor calibration tool (see Figure 8). The work cell layout can be completed adding environmental objects, such as the floor or the product fixtures. Collisions with all these objects are taken in account (see Figure 6).

Figure 5. Coverage planner

FibreMap FP7 608768

7

Figure 7. Rectangles representing the covered area.

To manage and setup the inspection process a Fibremap application has been developed. The application is a front end to help the user to configure all the parameters for the inspection system. When the application starts, a new inspection project must be created defining the 3D model of the product to be inspected. The Fibremap application contains an editor where a specific panel to set-up the parameters for the path planning has been developed (see Figure 5).

4. Coverage and path planning overview

At the heart of the robot path generation there is the coverage planning software that ensures that the whole part is fully scanned. This software module will take as input the 3D shape of a part and it will provide as output a path for the robot and a list of positions, called viewpoints, where images need to be taken to fully scan the part (excluding tight corners and other elements where image acquisition is infeasible). IT+Robotics put particular emphasis on the need to acquire overlapping images taken from a single surface element. The generation of overlapping images or viewpoints consists of several operations. The first step consists of covering the 3D model of the product with rectangles that represent the area imaged by a single camera frame. In this context, a rectangle is a planar entity placed in the 3D space (Figure 7 represents the final rectangles on the surface). The size of each rectangle is computed by camera parameters. We model the “FScan” sensor as a pinhole camera (see Figure 9), that describes what the sensor can see from a given

Figure 6. Workcell Editor collision checking feature.

FibreMap FP7 608768

8

viewpoint. This can be computed by considering the sensor size and focus distance. The viewpoints cannot be too far or to close from the desired product distance, otherwise the images will be out of focus.

Given the set of viewpoints that needs to be reached by the camera, a robot path is generated considering the following constraints:

• The robot kinematics. The geometry of the robot introduces constraints on the positions that can be reached. In other words, each viewpoint has to be reachable by the robot;

• The workspace. The viewpoint must avoid collisions with other objects in the workspace (e.g. product fixtures and the product itself). Collisions between robot and object, between sensor and robot, and between sensor and object must be taken into account.

The path computation is basically a Travelling Salesman Problem (TSP). TSP models the following problem: given a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city exactly once and returns to the

Figure 9. Pinhole camera model.

Figure 8. Fibremap calibration tool.

FibreMap FP7 608768

9

origin city? It is an NP-hard problem in combinatorial optimization, and it requires heuristics to obtain a sub optimal solution in polynomial time. The path planning algorithm tries to cover uniformly all the surfaces, both planar and non-planar, keeping the total number of viewpoints as low as possible. Such condition is optimal in a production environment, where scanning time has to be minimal. A set-coverage greedy procedure is performed for this purpose. At each iteration the position that fills the most yet uncovered area of the surface is chosen. The algorithm stops when the overlapping percentage covered by the last position chosen is greater than a threshold, called “Maximum allowed overlapping”. This threshold is a percentage relative to the maximum surface area that a position can cover. This parameter is connected to the one called “Ratio of intersection”, that describes the relative percentage of covered area shared between two consecutive camera frames. Tuning maximum allowed overlapping based on ratio of intersection leads to full coverage of the part.

Figure 10. Coverage planner changing the maximum allowed overlapping The greedy cost function is paired with two heuristics. The first heuristic, called “angle penalty” is calculated computing the angle between next position and the line connecting the two previous ones, trying to avoid too many changes in direction performed by the robot. The second heuristic is called “centroid penalty”: the centroid is computed at each

step of the path generation, and it’s the average of the positions already chosen. The centroid penalty increases when the distance with centroid increases. This heuristic, as shown in Figure 11, lead to a visible ordering of the path, that can avoid robot configuration changes and thus, wasting of time not accounted in the greedy strategy of Euclidean distance.

Figure 11 Path ordering without (left) and with (right) heuristic.

FibreMap FP7 608768

10

The trajectory generated with the simulator is then translated to the robot JSON data format and sent through the developed communication module between the path planning software and the path listener in the Linux PC as explained in section 2.

5. Robotic system results

Figure 12: Complex part scan: fibre orientations are color-coded and projected onto the surface of the CAD model. A carbon fibre pre-form with a size of about 0.7m² was measured using the robotic system. During the scanning about 10.000 single images were taking, of which sets of 8 images were combined for calculating the fibre orientation. Fibre orientations were then mapped to a 3D mesh in a dense grid. The grid size can be arbitrarily defined, depending on the needs of the application up to a limit of about 200 million single point. This upper limit is determined by the lateral resolution of the sensor. The standard deviation of the in-plane fibre orientation could be confirmed to be less than 1°, when performing multiple measurements of the same location.

6. Further information

FibreMap project video: http://youtu.be/wkqbG74II0M FibreMap web page: www.fibremap.eu Coordiantor Contact: Profactor GmbH, Austria

Dr. Christian Eitzinger; e-mail: christian.eitzinger@profactor.at