Online Image Analysis for Structural Experiments using Mobile Device and Internet of

Things Techniques

Chung-Ming Yang1 and Yuan-Sen Yang2

1) Team Leader, Chunghwa Telecom Co. Ltd, TAIWAN; Ph.D. Candidate, National Taipei University of Technology,

Taipei, TAIWAN.

2) Associate Professor, National Taipei University of Technology, Taipei, TAIWAN

Abstract:

Recent advancements on high-power hydraulic actuators, data acquisition performance, and digital control

technology allow earthquake engineering researchers to design and carry out larger and more complicated

structural experiments. These experiments are normally relatively expensive, more labor intensive, and require

larger payloads, higher hydraulic pressure for larger applied forces, larger and more complicated fixtures and

protection frames, leading to a relatively higher risk on experimental safety. An image based concrete crack

observation approach has been investigated by the authors, aiming to reduce the need of manual pen marking and

improve the experiment safety. It is implemented as a software tool, named ImPro Stereo, giving good

programming flexibility for customization. This tool is capable of not only estimating 3-dimensional displacements

and surface strain fields, but also presenting cracks that are thinner than what naked eyes can observe, and has

been employed by some reinforced concrete (RC) tests. However, its image analysis work were carried out offline,

required a few manual operations and hours or days to run image analysis before the results were presented, not

capable of responding results immediately for instant experimental controlling judgements.

This paper introduces the software design and implementation of a new generation of ImPro Stereo, aiming better

automation and higher computing performance for online analysis. In addition to showing basic flowchart and

formula, this paper presents the software framework on online image acquisition, camera calibration, fast image

rectification and projection analysis, and deformation, strain, and thin crack analysis. In addition, considering the

recent rapid advancement of camera equipped mobile devices, this paper also introduces how camera equipped

with mobile devices are employed in this approach, offering chance to control shutters and focal lengths, and

acquire images, camera vibrations, gyro data online wirelessly. Usage of latest Internet-of-Things communication

programming protocol employed in this approach is also introduced. This paper is then finalized with an

experimental plan of this approach to a coming large-scale RC experiment is also presented in this paper.

Keywords: Image analysis, earthquake engineering experiments, ImPro Stereo, online analysis.

1. INTRODUCTION

As the structural experiments are getting more complicated, destructive to sensors, and risky to experimental safety,

remote sensing technology, including laser scanning, optical tracking, and image analysis, has started to be

employed by researchers to not only reduce the sensor loss, acquire more data and information of experiments, but

also reduce the labors and improve the safety. Among the remote sensing technology, the image analysis is

probably one of the most flexible, customizable, programmable, and affordable for different types and scales of

structural experiments.

An image analysis software tool, named ImPro Stereo, has been developed and improved in the recent years (Yang

et al. 2012). It measures 3-dimensional displacements, surface strain fields, and concrete surface cracks of regions

of interests. Partcially powered by OpenCV (OpenCV, 2015), a widely used open source computer vision library,

and based on MATLAB development environment, ImPro Stereo is highly customizable. It was employed by

many earthquake engineering experiments recently to measure point movement (Patel et al., 2015), strain fields,

wind turbine vibration (Loh et al. 2016), concrete surface cracks (Yang et al. 2014; Yang et al. 2015a; Yang et al.

2015b), and surface retrofitting de-bonding observation.

While ImPro Stereo is flexible, and relatively versatile and customizable, however, it is gradually getting behind

the demanding measurement requirements. It requires to be capable of acquiring photos instantly while the

experiments are still running, responding the analyzed results instantly, camera access, better automated that

requires less user operations, without loss of versatility and customizability.

In terms of the computing speed requirement, the programming of time consuming image analysis need to be re-

coded. While the old ImPro Stereo employed OpenCV adapted with C++ as its analysis kernel for image template

matching and optical flow, the camera calibration and image rectification analysis and visualization was based on

MATLAB interpretive codes, which although is good for customization, is relatively slow and memory

1501

consumptive. Strain fields or crack analysis of a region of interests taken by a pair of 24-mega-pixel cameras

normally requires roughly one minute to analyze, which is not quite sufficient for online analysis purpose. More

OpenCV kernels need to be employed for calibration and rectification based on a C++ (on desktops) and Java (on

mobile devices) development platform.

(a) Foundation rocking analysis (b) Wind turbine blade vibration (c) Concrete shear crack analysis

Figure 1 Selected applications of ImPro Stereo on structural experiments

Online camera access, such as vibration monitoring and focal length control, requires cameras to be programmable.

Cameras need to be programmable for remote data accessing and control. Although industrial smart cameras is

rather programmable, however they are expensive, with relatively low resolution, and limited by manufacture-

defined languages. Recent mobile devices, (i.e., Android powered ones in this work) are more flexible and versatile.

Programming mobile devices involve awareness of event-driven, threading, call-back functions, leading to data

synchronization issues that will be discussed later.

Figure 2 Configuration of Experiment and Image Analysis System

0 1 2 3 4 5 6 7 8 9 100

2000

4000

6000

8000

10000

12000

14000

Frequency (Hz)

|Y(f)

|FFT

4.1 Hz(tower freq.)

2.6 Hz(damaged blade)

Actuator Controller

Program-controlled Cameras

(equipped withOpenCV kernel and IOT networking)

Single-boardComputer

(equipped withIOT networking)

Image analysis Computer(equipped with

ImPro Stereo, OpenCV kernel, and IOT networking)

Analog action signal

Wirelessaction signal

Wirelessdata transfer

Control commands

Image Analysis System

Specimen in Lab.

Analog action signal

Conventional data logger

Experimental Control & Measurement

Sensing data

1502

Improvement of automation on image analysis requires minimal user operations of image analysis while an

experiment is still running. Old ImPro Stereo required much user operations in stages of camera calibration, file

manipulations, control points definitions, and many different parameter setting at different stages of image analysis.

To minimize user operations online, OpenCV calibration functionality, which highly automatically detects grids

of calibration corners is employed. Data driven mechanism is implemented in the new ImPro Stereo so that image

analysis is automatically triggered when new photos come during experiments.

Photos, instant preview images, camera status information and camera control signals are automatically transferred

among cameras, devices, and new ImPro Stereo online through Internet-of-Things (IOT) protocol (i.e., Alljoyn

protocol led by Qualcomm, Inc. and widely supported by manufacturers such as Foxconn Group) wirelessly

(Allseen Alliance Inc., 2015). The IOT protocol not only avoids sophisticated details of Internet socket operations

in programming, but also ensures better programming maintenance, well optimized performance and hardware

compatibility. Microcomputers such as Raspberry Pi series that is highly portable, low-power consuming,

equipped with wireless and IOT supports, capable of converting experimental analog signal to wireless camera

control.

2. ANALYSIS METHOD AND IMPLEMENTATION

One of the important features of the ImPro Stereo is its precise deformation analysis based on image metric

rectification. Instead of massively running millions of image correlation, the ImPro Stereo adopts a metric

rectification step, proposed by Yang et al. (2012), that remaps a rectified image of the region of interests before

running deformation analysis. This method not only shortens computing time on massive image correlation, but

also significantly reduces the noise induced by the sensitive errors of stereo triangulation. Through metric

rectification, the strain field precision can reach 0.02 pixels, that is, about 1/200,000of the width or height of the

region of interests (assuming the camera resolution is about 4000 by 4000). Due to the space constraint, details are

not presented here. Figure xx shows a rough flowchart of the analysis method adopted by the ImPro Stereo.

Figure 3 Basic Flowchart of ImPro Stereo Image Analysis

Image tracking is one of the most important functions in image measurement. Given a sub-image template of the

point to track, image tracking determines the location of the point in the searched image. OpenCV provides a

template matching function to do this job but the precision is 1 pixel without any sub-pixel analysis. In addition,

it is not rotational invariant, which would fail to track a rotated object, such as a target on a rotating wind turbine

blade. Old ImPro Stereo implemented a two-layer template matching analysis (Yang et al. 2012) but is time

consuming. This work reduces its computing time without losing precision by implementing a multi-layer rotatable

Calibration Photos

Calibration

3-D Positioning of Control Points

(Stereo Triangulation)

Surface Parameters

Metric Rectification

Deformation Analysis

Calibrated parameters

Left Right

Experimental Photos

Left Right

3-D Positions of Control Points

Surface Shapes

P1P2

P3

P4

O

CRegion of

Interest (ROI)

Control point

Rectified Images

Displacements, Strains and Cracks

Next step

Procedure

Data or Images

Data flow

Procedure flow

1503

template matching analysis. In addition, it is rather rotational invariant because it also searches different rotating

angles in addition to different x and y locations.

Here presents a preliminary performance test of the multi-layer rotatable template matching analysis

implementation. Figure 4 shows the testing template, a 20-by-20-pixel of a standard corner, which is widely used

for image positioning. The searched images are rotated corners with artificially added noise. In each test, both

template and search images are automatically resized in each layer. Number of layers are determined automatically,

and were seven layers in these cases. Even after artificial noises, the positioning and rotation analyzed errors were

less than 0.06 pixels and 1.4 degrees, respectively. Results of without noise, which were not listed here due to

limited space, showed that positioning and rotation errors were less than 0.01 pixels and 1 degree, respectively.

The CPU time is much reduced from 1 second using two-layer template matching in old ImPro Stereo to 0.2

seconds using multi-layer rotatable analysis, not to mention that the latter one additionally analyzed rotations. The

improvement is not only contributed by the implementation from migrating from C++/MATLAB hybrid to pure

C++, but also the employment of multi-layer scheme.

(a) template (b) 0-degree (c) 20-degree (d) 40-degree (e) 60-degree (f) 80-degree

Figure 4 Images for multi-layer rotatable template matching performance test

Table 1 Results of multi-layer rotatable template matching performance test

0-degree 20-degree 40-degree 60-degree 80-degree

X Error (pixels) 0.012 0.015 0.004 0.015 0.064

Y Error (pixels) 0.015 0.025 0.030 0.009 0.062

Rotation Error

(deg.) (radian)

0.9 degrees or

0.016 radians

1.4 degrees or

0.025 radians

0.2 degrees or

0.004 radians

1.0 degrees or

0.018 radians

0.1 degrees or

0.001 radians

CPU Time (sec.) 0.20 0.11 0.09 0.09 0.14

Metric rectification is the key process that ImPro Stereo reaches high precision when measuring surface

deformation, however, its long computing time is one of the obstacles to step forward online analysis. In the old

ImPro Stereo, it normally requires about one minute to generate a rectified image, which spends most of the

computing time in image analysis. Rectification requires computing on generating a dense grid of 3-dimensional

points, projecting to image points, remapping and interpolating to three-channel color intensities. These processing

were re-implemented in C++ with help of OpenCV functions. The computing time reduces by more than 95%,

from about 60 seconds to less than 2 seconds. With further improvement by using multi-core computing using

OpenMP, the actual wall time can be further reduced to less than 0.5 seconds, making online analysis feasible.

The testing computer is an ASUS G56JR laptop equipped with Intel i7 4700HQ CPU and 16 GB main memory.

3. IMAGE ACQUISITION AND NETWORKING

Migrating from offline to online image analysis requires not only significant improvement on computing

performance, but also the advancement on image acquisition and networking. Image acquisition and networking

are relatively small issues in offline analysis. Offline analysis is shown in Fig. 5(a). The device acquires and store

images during experiment. Then, after experiment, we retrieve images from device to computer and do the analysis

on computer. The disadvantage of this method is the experiment had already finished when we analyzed images.

We do not know if the images are qualified for analyzing. For example the device may be moved or affected by

vibration during experiment. Another disadvantage is this method is not suitable for real time monitoring system

which is required in many cases. In order to solve these problems, we need a new method which can analyze

images as soon as possible when device acquires it. That is the main concept of online analysis shown in Fig. 5(b).

When the device acquires images, it should send image directly to who need it. Then, the other device or computer

can perform its function without waiting.

1504

Offline Analysis

After ExperimentDuring Experiment

Acquire Images(device)

Store Images(device)

Retrieve Images(device to computer)

Analyze Images(computer)

(a)

Online Analysis

During Experiment

Analyze Images(computer)

Acquire Images(device)

Store Images (device and computer)

(b)

Figure 5 Concept of (a) Offline Analysis and (b) Online Analysis

The programming pattern of device data acquisition is a more asynchronous observer-pattern or listener-pattern.

Comparing to a more conventional pattern that directly and synchronously calls sub-programs that acquires data

(Fig. 6(a)), the asynchronous pattern is widely used for modern mobile device systems and many graphical user

interface frameworks (Fig. 6(b)). While the conventional synchronous pattern is easier to understand and use to

design programming framework, it is not used for modern mobile device programming framework. The modern

pattern for data acquisition is believed to be more reliable and flexible for a multi-tasking system that hardware

resource can be accessed by multiple tasks. It also prevents system halt due to hardware issue. However, it can

induce an uncertainty level of synchronization of data, especially when we acquire multiple types of data. In this

work we acquire data from camera and accelerometers (Fig. 6(b)). In the modern pattern, we register a callback

function that manipulate the acquired data or images, and the callback function is called automatically by the

system of the device. If we register two or more callback functions for different types of sensors, which is the case

in this work, we do not exactly know which callback function is called first and it is not easy to measure the time

difference between them. Using timestamps could partially solve the problem, however, if the sampling rates are

higher than the timestamp precision, a certain level of uncertainty still exists. While data acquisition operations

are not directly and synchronously triggered by our program, we have to accept the uncertainty on the

synchronization between photos and vibration data.

(a) A conventional pattern for data acquisition (b) A more modern pattern for data acquisition (b)

Figure 6 Conventional and more modern programming patterns for device data acquisition

The main requirement of the online analysis method is that the devices and computers should have ability to

communicate with each other during an experiment. It is the same as the main concept of the Internet of Things

(IoT). The IoT technology aims to construct a network environment that is suitable for small devices to find and

communicate with each other. IoT also allows devices to be controlled remotely across network infrastructure.

There are several framework like Alljoyn (Allseen Alliance Inc., 2015) IoTivity can help people develop IoT

software. Only Alljoyn was available at the time we designed our online analysis method so we base on Alljoyn

Main Camera Accelerometer

3. TakePicture

4. Return Image

1. GetAcceleration

2. Return Acceleration

Main Camera Accelerometer

3. TakePicture(PictureCallback)

2. Return Acceleration

1. RegisterSensorEventListener

SensorEventListenerPictureCallback

Loop

4. Return Image

1505

to implement our online analysis method.

AllJoyn (Allseen Alliance, Inc., 2015) is a collaborative open-source software framework that makes it easy for

developers to write applications that can discover nearby devices, and communicate with each other directly

regardless of brands, categories, transports, and OSes without the need of the cloud. The AllJoyn framework is

extremely flexible with many features to help make the vision of the Internet of Things come to life.

Global Space

Node 1

Node 3

Node 2

Node 4

Cha

nnel

3Channel 2C

hann

el 1

Figure 7 Concept of network communication design implemented in this work

We constructed a virtual environment called Global Space, which maps to a physical experimental environment.

Every participant in the physical experimental environment, e.g., cameras and computers, is a node in the Global

Space. Each node may contain several channels as shown in Figure 7. Node will advertise itself through global

space when it joins to the Global Space. If it can provide image, like camera, it will register a channel to deliver

image to other nodes. Any other node who needs the image from this channel can acquire this channel. Every time

the new image is generated it will write through channel and anyone who has already acquired this channel will

receive this new image. Then, by using observer pattern it will notify node there is a new image ready to be

processed. We can process images in parallel by using this mechanism. For example one node store data and the

other calculate strain filed.

We chose Android as the operating system of mobile devices to acquire images because it is widely adopted by

many different types of camera equipped mobile devices, and was the only mobile systems officially supported by

OpenCV at the time we were working on this topic. An Android device has several useful features for our design.

First it can be programmed; we can implement our design by writing app on it. Second it has ability to communicate

through network. Third it has accelerometer can detect accelerations. We can record accelerations when images

are taken which will help us to identify if the image is taken on the stable situation. In addition these useful features,

there are some mechanism android used need to be noted; when you command it to do something, it does not

usually return result directly. You need to register listener to receive result so we can’t actually control when they

do the command, although it will do the command as soon as possible. In our application we use timestamp to

verify images acquired in the same experiment step. If the difference of timestamp between them is less than our

threshold, we considered synchronized.

This work implemented an OpenCV-powered mobile program that captures either preview-mode information or

high-resolution information, and transfers to the PC based analysis program. In the preview-mode, high-frame-

rate with low-resolution photos and camera tri-axial vibration (acceleration) time sequential data were captured

and transferred instantly, continuously, and wirelessly to the PC. While an action signal is received (which is sent

by a single-board microcomputer under Alljoyn protocol), a high-resolution photo with its corresponding tri-axial

vibration time sequential data are sent. High resolution photos are used for analyzing displacements, strain fields,

or cracks, and the vibration information is used for checking the stability of the cameras. If the vibration is higher

than a certain level, it implies that the photos can be taken while cameras are vibrating, and the photos can be

considered invalid for image analysis. Figure 8 shows a software test of the mobile device program that ran on an

Android camera and a PC based data viewer (photos and tri-axial accelerations). The PC represents a computer

that will run the new ImPro Stereo to complete computationally intensive image analysis jobs. The image that was

taken in the software test was a hypothetical image, not an actual structure. The test preliminarily verified the

reliability and performance of remote image acquisition, data acquisition, and network performance.

1506

(a) Mobile device running data acquisition and networking; (b) Networked PC communicating with mobile device

Figure 8 Laboratory Testing of Image Acquisition and Networking Programs

4. SUMMARY

Aiming to improve image analysis service from offline analysis to online analysis that is capable to provide

information for instant understanding on specimen health status or experimental controlling judgement, this work

re-organized and re-implemented an existing image analysis tool ImPro Stereo. The new program framework is

preferable to online data acquisition and instant image analysis. Program platform is migrated from

MATLAB/C++ hybrid to C++ (host side) and Android/Java (device side). More OpenCV functions are employed

for better computing performance. A multi-layer rotatable templated matching is implemented in this work for

faster and more rotation invariant tracking. Image rectification computing time is reduced by more than 95%, that

is, achieving a speed up greater than 20. Mobile devices are employed and their corresponding data acquisition

program is developed, which not only acquires photos but also camera vibration information. The Internet-of-

Things communication protocol is employed for better programming maintenance and future compatibility

between devices. A single-board microcomputer is programmed to convert analog signal generated by

experimental facility to camera control commands through wireless digital signal based on the IoT protocol.

The integration of the improvement aforementioned is an experimental image analysis tool that is capable of

efficient online analysis. It is capable of providing instant analyzed information for better understanding of the

health status of specimen and for experimental control judgement. At the time this paper is being written, this tool

is close to completion and will be used for observing and analyzing concrete surface shear crack development of

a large-scale RC experiment.

ACKNOWLEDGMENTS

The authors would like to thank Moore Professor Thomas Hsu of University of Houston, Dr. Chiun-lin Wu, Dr.

Y. C. Chen, Dr. H. C. Yang, Dr. H. J. Lu, and Dr. C. C. Chang of National Center for Research on Earthquake

Engineering for their supports on the reinforced concrete experiments. This work is partially supported by a

research project from Ministry of Science and Technology, Taiwan (No. MOST-104-2625-M-027-001).

REFERENCES

Allseen Alliance Inc. (2015) AllJoyn Framework Retrived from Allseen Alliance website:

http://www.allseenalliance.org/framework, accessed on August 1, 2015.

Loh, C. H., Huang, Y. T., Hsiung, W. Y., Yang, Y. S., and Loh, K. J. (2016). Vibration-Based Identification of

Rotating Blades Using Rodrigues’ Rotation Formula from A 3-D Measurement, Wind and Structures,

accepted to publish.

OpenCV. (2015). OpenCV. Retrieved from OpenCV website: http://opencv.org/, accessed on August 1, 2015.

Patel, V. J., Van, B. C., Henry, R. C., and Clifton, G. C. (2015). Effect of Reinforcing Steel Bond on the

Cracking Behaviour of Lightly Reinforced Concrete Members. Construction and Building Materials,

96, 238-247.

Yang, Y. S., Huang, C. W., & Wu, C. L. (2012). A Simple Image-based Strain Measurement Method for Measuring

the Strain Fields in an RC-wall Experiment. Earthquake Engineering & Structural Dynamics, 41(1), 1-

17.

Yang, Y. S., Yang, C. M., & Huang, C. W. (2015a). Thin Crack Observation in a Reinforced Concrete Bridge

Pier Test using Image Processing and Analysis. Advances in Engineering Software, 83, 99-108.

Yang, Y. S., Yang, H. C., Chen, Y. C., Lu, H. J., Chang, C. C., and Hsu, T. T. C. (2015b). Image Analysis for

Crack Observation of a Cylinder RC Structure Cyclic Test, Proceedings of the 6th International

Conference on Advances in Experimental Structural Engineering, University of Illinois at Urbana

Champaign, IL, USA.

1507