AI in Industry

94 AI MAGAZINE

Testing software systems is a difficult cognitive task thatrequires human skills to generate test scenarios and topredict the expected behaviors of a system. Moreover,

coming up with possible failure scenarios that may arise dur-ing system operation and determining triggers for these cas-es is a major challenge.

ABB Robotics’ painting robots are advanced distributedsystems that must be thoroughly tested before they areshipped to customers, mostly from the automotive andavionics domains. Improving the overall quality of paintingrobots has always been of utmost importance because it isviewed as a competitive advantage in the market for indus-trial robots — an error in the control system of these robots,uncovered in the field, can have serious economic conse-quences. Among the many possible failure sources are those

Copyright © 2017, Association for the Advancement of Artificial Intelligence. All rights reserved. ISSN 0738-4602

Deploying Constraint Programming for Testing ABB’s Painting Robots

Morten Mossige, Arnaud Gotlieb, Hein Meling

n This report explores the use of con-straint programming for the validationof ABB Robotics’ painting robots.

AI in Industry

SUMMER 2017 95

related to timing the triggering of different subsys-tems. For example, an overlap-error scenario canoccur when the robot’s speed increases and switch-ing between different paint spray patterns occurs toorapidly. In principle, the control system shoulddetect overlap scenarios, send an appropriate errormessage, and possibly shut down the system. How-ever, it may happen that the robot does not work asexpected, for example due to bugs in the design orimplementation of the control system. Creating testscenarios targeted at finding such subtle faults is non-trivial.

Faced with this challenge, researchers and engi-neers from Certus, a Norwegian research-based inno-vation center, have teamed up to explore the usage ofconstraint programming (CP) for the validation ofABB’s painting robots. Constraint-satisfaction tech-niques over finite domains were selected because oftheir versatility in dealing with heterogeneous con-straints derived from the design and implementationof ABB’s control system for paint robots. We used acombination of constraint propagation with differ-ent filtering consistencies and dedicated searchheuristics. With these techniques it was possible topropose a cost-effective solution for test case genera-tion for validating the control systems of ABB’s paint-ing robots (Mossige, Gotlieb, and Meling 2015;Mossige, Gotliev, and Meling 2014a, 2014b). Cur-rently, these AI techniques are commonly supportedby modern CP solvers that ease their adoption inindustrial contexts. However, using a constraintsolver to generate tests can be time consuming,which is at odds with the need to run both test gen-eration and test execution as part of a continuousintegration cycle. Indeed, continuous integration is asoftware engineering practice, where the result of testexecution, following a source code change, should bereported back to the developer quickly.

Despite the strong technical culture of ABB Robot-ics and the availability of excellent textbooks,deploying CP was not easy, as there are almost noguidelines on how to model with constraints andhow to integrate CP in an industrial software pro-duction process. However, we managed to developsuch a model and to deploy it at ABB Robotics byadding the constraint-solving process to the contin-uous integration process. The model has now beenpart of ABB’s continuous integration process for morethan 24 months of daily operations. After having col-lected data about its bug-finding capabilities and effi-ciency, we perform a thorough analysis of its benefitsand drawbacks, which led us to draw some lessonslearned from this experience.

Bug-Finding Capabilities of ABB’s CP Model

For the purpose of validating the CP model itself,known bugs were reintroduced in the paint control

systems to examine the capabilities of the model tofind them. After some tuning, the model was able togenerate test scenarios that could find all the re-intro-duced bugs. This was considered a significant successand justified the continuation of the research andinnovation activities. Next, immediately after theintegration of the model in the software productionsystem, several unknown, but subtle bugs were foundrelated to the timing aspects of the paint control sys-tems. However, these bugs were classified as noncrit-ical, as they represented very unlikely scenarios.Upon further analysis, we observed that these bugshad been present in the control systems for severalyears without any significant consequences. Theywere corrected and the generated test scenarios wereadded to the testing process. However, although thebug-finding capabilities of the model had beenstrongly validated, ABB believed that its ability touncover mainstream bugs still remained to bedemonstrated.

Deploying CP in Industrial Software Validation

The validation of paint robots involves a fair amountof manual work, which is also error prone by nature.Therefore, deploying CP as a way to automate someparts of the process was welcomed by validation engi-neers and perceived as a way to strengthen the vali-dation process. However, CP also comes with somechallenges. The adoption of CP requires training, andthe maintenance of the CP model without expertassistance is difficult, especially since the automati-cally generated test scenarios are difficult to under-stand for untrained engineers. Tuning the model andits solving parameters, especially when optimizationprocedures are used, turned out to be reserved forexperts only. Validation engineers are usually skepti-cal about tools that produce results that are incom-prehensible to them, and almost impossible to com-pute by hand. To reduce the risks of rejection,internal training sessions on CP were organized andvarious front ends were built to help the engineersmanage the complexity of the generated test scenar-ios.

Return on Investment on Using CPTo compute the return on investment, one can meas-ure the number of uncovered bugs with and withoutthe CP model or compare the human effort requiredin both cases. However, for us, another more impor-tant factor was evaluating the possibly increased con-fidence of the engineers in the validation process —a factor almost impossible to quantify. After havingdeployed CP, we observed an increased appetiteamong engineers to refactor code covered by the CP-generated tests. Such refactoring is needed, but oftendeferred due to code complexity, which over time

AI in Industry

96 AI MAGAZINE

can result in technical debt for the system. Upon fur-ther analysis, we understood that this came fromtheir increased confidence in the validation processas it acted as a safeguard to detect undesirable sideeffects.

Improving the overall quality of painting robotshas always been of utmost importance, because it isviewed as a competitive advantage. For this study, wechose to automate tests that covers the most criticalaspects of the system, as well as tests that were themost laborious to conduct manually.

While the goal of the project was not aimed at costsavings, we did observe that validation engineers arenow relieved to focus on other tasks that have yet tobe automated. In the long run, however, we expect todetect more bugs before shipping a product, ratherthan detecting bugs at customer premises, wherethey are typically much costlier to fix.

The deployment of CP as a part of a continuousintegration process at ABB Robotics, Norway, wasconsidered to be a success. It led the company toengage in an ambitious deployment plan at theworldwide level.

ReferencesMossige, M.; Gotlieb, A.; and Meling, H. 2015. Testing RobotControllers Using Constraint Programming and Continu-ous Integration. Information and Software Technolog, 57(Jan-uary): 169–185.

Mossige, M.; Gotlieb, A.; and Meling, H. 2014a. TestingRobotized Paint System Using Constraint Programming: AnIndustrial Case Study. In Testing Software and Systems: 26thIFIP WG 6.1 International Conference Proceedings, LectureNotes in Computer Science 8763. Berlin: Springer.

Mossige, M.; Gotlieb, A.; and Meling, H. 2014b. Using CP inAutomatic Test Generation for ABB Robotics’ Paint ControlSystem. In Principles and Practice of Constraint Programming:20th International Conference Proceedings, Lecture Notes inComputer Science 865, 25–41. Berlin: Springer.

Morten Mossige is a principal engineer at ABB Robotics inBryne, Norway. He is also an adjunct associate professor atthe Department of Electrical Engineering and Computer Sci-ence in the University of Stavanger, Norway.

Arnaud Gotlieb is leader of the Certus research-based inno-vation center, Simula Research Laboratory, Fornebu, Nor-way.

Hein Meling is a professor in the Department of ElectricalEngineering and Computer Science at the University of Sta-vanger, Norway.

Important Update!AAAI-18 ConferenceDates Have Shifted!

February 2-7, 2018 n New Orleans, Louisiana, USA

Update your calendar! AAAI-18 will now be heldFriday, February 2 – Wednesday, February 7, a shiftof two days earlier. This shift in dates will enableus to continue with our traditional pattern of hav-ing the workshops and tutorials prior to the com-mencement of the main technical program. Theconference venue remain the same at the Hilton New Orleans Riverside.

Please mark the following dates on your calendar:July 1 – September 8, 2017: Authors register on theAAAI web site

September 8, 2017: Electronic abstracts due at 11:59PM UTC-10 (midnight Hawaii)

September 11, 2017: Electronic papers due at 11:59PM UTC-10 (midnight Hawaii)

October 16-19, 2017: Author feedback about initialreviews

November 9, 2017: Notification of acceptance orrejection

November 21, 2017: Camera-ready copy due at 5:00PM PDT (California)