summary for cife seed proposals for academic year 2018-19 ... · the position of the mobile robots...

Summary for CIFE Seed Proposals for Academic Year 2018-19

Proposal number:

Proposal title: A framework for dynamically updating building information models using mobile robots and computer vision

Principal investigator(s)1 and department(s):

Kincho Law

Research staff: Max Ferguson, PhD Candidate

Total funds requested: $

Project URL for continuation proposals

http://

Project objectives addressed by proposal2

Buildable, Operable (If extended +Usable)

Expected time horizon < 2 to 5 years >

Type of innovation Breakthrough

Abstract (up to 150 words)

The problem: Automation in construction and facility management could significantly improve efficiency and productivity throughout the lifecycle of modern facilities. However, for robots and autonomous systems to operate in dynamic and unstructured environments such as a construction sites or medical facilities, they must be able to infer or obtain a semantic spatiotemporal model of the environment. The proposed solution: We will develop a framework for dynamically updating a building information model using recent advancements in computer vision. The framework will enable sensor data from each robot to be combined with known building geometry to form a rich semantic model of the environment. In this way, it will be possible to leverage the collective knowledge of multiple robots and building sensors to make better global decisions. The proposed research approach: Our research efforts will be directed towards detecting common objects in the construction site or modern facility (such as safety cones or furniture) and placing those objects in a model of facility. Contrary to previous approaches, we will attempt to update the position of the relevant objects, in near real-time.

<K. H. Law > < Dynamically Updating Building Information Models> 2

Engineering or Business Problem Describe a case example or situation that motivates the proposed research, preferably based on a real scenario. Quantify the problem and its impact as much as possible in terms of the usability, buildability, operability, or sustainability criteria applicable to the problem and the type(s) of facilities to which your proposed research applies. Explain the problem with an annotated figure or photo. Automation in construction, maintenance and facility management promises to bring significant productivity gains to a number of industries. However, for robots to operate in dynamic and unstructured environments such as construction sites or medical facilities, they must be able to infer or obtain a semantic spatiotemporal model of the environment. Such a model may include static information about the facility, such as the position of door and walls. To facilitate safe and efficient automation, the model should also contain dynamic information about the environment, such as the position of other robots in the facility. The position of movable objects, like furniture or safety cones, might also be of interest. During construction, the model can be used to enable greater automation, as well as providing a real-time source of information for construction managers and related professionals. In the operations phase, the spatiotemporal information can be used to ensure that automated processes are operating in a safe and efficient manner. Hence, real-time spatiotemporal semantic information can be beneficial throughout the majority of a facility’s lifecycle phases. Based on the above, such information can influence the following aspects of a facility: (a) Buildability, by facilitating automated construction processes (b) Operability, by enabling the safer operation and efficient management of mobile robots in the facility. (c) Sustainability, by automating the collection of data for decision making related to improving productivity during construction, and efficiency in facility operation. Additionally, real-time semantic information about the building can be used for controlling and inspecting productivity, quality, and safety, which alleviates risks and opens new pathways for owners, operators, designers and builders. Construction is one of the largest industries and contributes to about 10 % of the gross national product (GNP) in industrialized countries [1]. Hence, the performance of the construction industry has a significant impact on national economies. Similarly, automation in construction provides a potential solution for increased productivity and safety in construction. Automation in facility management and operations also presents an opportunity for efficiency gains. The facility management market is expected to grow from USD 34.65 Billion globally in 2018 to USD 59.33 Billion by 2023. Building managers are likely to look towards automation as the market grows. In the area of facility management, security robots can be deployed to increase security in high-risk areas like parking garages [1] and logistic robots can be deployed to move supplies around large facilities, such as hospitals or airports [2]. However, for these automated systems to work safely and efficiently, they must have access to, or be capable of generating a semantic model of the facility.

1 The PI(s) must be academic council member(s) at Stanford. 2 For this and the next points, delete the answers that don’t apply to your proposal.


According to the US National Building Information Model Standard Project Committee “Building Information Modeling (BIM) is a digital representation of physical and functional characteristics of a facility. A building information model is a shared knowledge resource of information about a facility forming a reliable basis for decisions during its life-cycle” [9]. Currently, building information models tend to represent the building at fixed points in time, either during or at the completion of the construction process. With the absence of real-time data, BIMs cannot form a reliable basis for automated decision making processes. Bridging the gap between real-time systems and building information models is critical to advancing automation in construction and facility management. Theoretical and Practical Points of Departure Describe relevant theory including concepts and methods developed at CIFE and relevant research at other academic institutions. Explain current industry practice, including topics such as management methods, workflows, commercial tools, and performance against the criteria explained in the “Engineering or Business Problem” enabled by current concepts and methods. Motivated by the limitations of current practice, research efforts regarding automation in construction have been directed towards the problem of automatically extracting semantic information from 2D images [3]–[6] and 3D point clouds [7]–[10]. Recent advances in deep learning have enabled semantic segmentation of large point cloud datasets [9]. However, the process of capturing these point clouds remains relatively slow, often taking hours or days. On the other hand, the collection and processing of 2D images tends to be much faster. Modern convolutional neural networks can extract semantic information from images at between 5-50 frames per second [11], [12]. There has been some interest in updating IFC models using semantic input data [13]. However, the task of updating BIM models using 2D images in is yet to be explored extensively by the research community.

IncreasinglevelofsemanticinformationDecreasingrateofchangeofinformation

BIMModelUpdatedmanuallybyhumanoperators

PointCloudCollectedatregularintervalsbyhumanoperatorsorrobots

ImagedataPotentiallycollectedconstantlybymobilerobots

?


The proposed research project builds on the success of Building Information Modelling (BIM). BIM is a process involving the generation and management of digital representations of physical and functional characteristics of places. BIM is a well-established field, both from a research and an industry perspective. Commercial tools such as Revit, ArchiCAD, and Vectorworks are commonly used to create, update and edit building models. However, these tools are not designed to support automated real-time updates from robots or sensors in the building. In real-time BIM, models are constantly updated using some form of sensor data. Researchers have proposed using real-time BIM to assist crane navigation during building construction [14]. A good discussion of real-time BIM is provided by Hwang and Liu, but they emphasize that the integration between real-time systems and BIM remains very low [15]. Some recent commercial offerings from architecture, engineering and construction (AEC) software vendors lean towards the integration of real-time data with BIM [16]. However, there does not appear to be any commercial offerings which are tailored to low-latency real-time building information modelling. The concept of automatically creating environment maps has been explored extensively by the robotics community. Perhaps the most noteworthy technique is Simultaneous Localization and Mapping (SLAM), which has attracted immense attention in the mobile robotics literature. SLAM is the computational problem of constructing a map of an unknown environment while simultaneously keeping track of an agent's location within it [17]. A number of open source tools, such as Hector SLAM [18] and Google Cartographer [19] are available for robot localization and mapping. While SLAM is useful for robot navigation, it does not provide the level of detailed sematic information about the built environment required for automating tasks in building construction, operation, or maintenance; A map generated using SLAM does not indicate what types of objects (such as furniture, safety equipment, or construction materials) are present in the environment. From the above, it appears there is a lack of end-to-end methods that address the problem of automatically updating a spatiotemporal model of the environment with semantic information from building sensors or mobile robots. Moreover, the potential impacts on the industry from obtaining such knowledge have not been quantified or fully explored beyond intuitions arising from evident applications. Research Methods and Work Plan Describe how you propose to carry out the research, the theoretical framework you will build on, and the specific kinds of work you plan to do. Explain how these follow from your theoretical and practical points of departure. Discuss observations of industry practice and the modeling and validation tasks you plan to undertake in a laboratory or industry setting. Use annotated diagrams and photos to explain your methods.

The planned future work can be organized into three main categories: developing a computer vision algorithm that can identify common objects in the facility or construction site; implementing a framework for real-time 3D building information model updates, and validating the proposed framework with laboratory tests. The main components of the framework can be seen in Figure 1.

1. Development of a computer vision algorithm to identify common objects. In this stage, we will build on recent advances in computer vision to develop an algorithm to detect the type and position of common building objects. Depending on feedback from industry partners


and the Technical Advisory Committee, we will focus on detecting common construction site objects, such as safety cones and signs, or common building objects such as furniture.

2. Development of a framework for real-time building information models. In this framework, the position of the mobile robots will be constantly broadcasted to the building information model. Movable objects, such as furniture or construction cones, will be added to the building information model, as they are discovered by the mobile robots.

3. Validation of the proposed framework using laboratory and facility-scale tests. To validate the proposed framework, we will conduct a number of real-world tests. A mobile robot will be built and fitted with a high-performance embedded computing device, capable of running the computer vision algorithm. The computer vision algorithm will be deployed to the mobile robot with the goal of detecting object of interest, in real-time. These objects will be dynamically added to a digital building model, using the proposed framework. Experimental tests will first be conducted in a laboratory environment, and subsequently conducted in a large-scale facility such as the Y2E2 building.

Figure 1. Flow of information from mobile robots to the building information model. Research will focus on developing the computer vision

module, as well as the overall framework.

Expected Results: Findings, Contributions, and Impact on Practice Summarize the anticipated results of the proposed research. What findings do you expect to develop? For whom? What theoretical contributions do you plan to formalize? If successfully translated to practice, what impacts do you expect? Who will benefit from these impacts? When? The biggest anticipated outcome of this research is a novel and elegant dynamic building information framework that will better facilitate bidirectional communication between mobile robots and building information models. This framework will be an ideological extension of the traditional BIM ontology, designed to better support automation in the construction site and modern facility. We expect that this contribution will be of interest to many parties in both industry and academia. The framework will provide an ideological platform for future research in real-time building information modelling. The framework will likely be of immediate interest to technology

Building Information Model

Building geometry

Data ingestion layer

Mobile Robot

Computer Vision

Locations of safety cones, furniture, etc

Mobile Robot

Computer Vision


companies in the AEC space, which are working on similar problems. Looking further ahead, the framework represents a step towards increased automation on the construction site and in the modern facility. If successful, the secondary outcome will be a computer vision algorithm capable classifying relevant objects in 2D images, and approximately locating those objects in 3D space. This algorithm will build on recent advances in computer vision. While not revolutionary, this algorithm may be of interest to computer vision researchers and industry professionals who are interested in real-time localization of objects, with application to construction site or facility management. Industry Involvement This proposal was prepared in collaboration with Doxel (R. Singh). We hope that Doxel will continue provide expertise and guidance as the project progresses. Autodesk (J. Rowe) has also been contacted and will provide support and knowledge in using Revit® to generate the static geometry for the dynamic building model. Currently, dynamic building information modelling is not a robust capability in the Autodesk suite of products. Additionally, we look to reach across CEE and bring in the resources of other research centers. We hope this is the first of numerous future collaborations between these, and other, research centers Research Milestones and Risks The following four milestones, which correspond with the research outline described above, will be used to measure the progress of the proposed research:

1. Development of a computer vision algorithm to locate common building objects, such furniture or construction safety cones

2. Extension of the existing 2D dynamic building information model support 3D geometry, and real-time geometry updates

3. Validation of the dynamic building information framework and computer vision algorithm,

using a mobile robot in the Y2E2 building

4. Submission of paper to peer-reviewed journal The risks are primarily associated with the inter-dependency of different milestones; for example, the validation of building information framework is affected by the accuracy and degree of maturity of the object recognition algorithm. To mitigate such risks, multiple strategies will be employed including:

• Researchers will focus on developing the different components of the system in parallel, relying on valid sets of assumptions about the structure of the input data at each milestone

• Some of the previously developed algorithms for 3D object reconstruction \cite and mobile robot control \cite will be used to streamline the development of the proposed solution.


Other risks include the ability to deploy a computer vision algorithm on an embedded system. Fortunately, a prototype device has been built and tested to ensure that the relevant computer vision algorithms can be run on the mobile robot. Next Steps Discuss your plans for obtaining external continuation funding for the proposed work. The proposed work may contribute to productivity gains during the construction and operation phase of the building lifecycle, and hence it will also be of interest to stakeholders in construction, facility management, operations, and maintenance. We expect that the framework will be particularly applicable for stakeholders who are interested in using robots to automate construction, operations, or maintenance. Future work will explore how the dynamic building information model can be used with artificial intelligence techniques to automate the management of mobile robots in the building. Budget On the last page of the proposal, insert the budget obtained from your research administrator.

summary for cife seed proposals for academic year 2018-19 ... · the position of the mobile robots...

Documents