An overview of the self-organizing map algorithm, on which the papers in this issue are based, is presented in this article.

The Self-Organizing Map

Automatic detection of pavement cracks is an important task in transportation maintenance for driving safety assurance. However, it remains a challenging task due to the intensity inhomogeneity of cracks and complexity of the background, e.g., the low contrast with surrounding pavement and possible shadows with similar intensity. Inspired by recent success on applying deep learning to computer vision and medical problems, a deep-learning based method for crack detection is proposed in this paper. A supervised deep convolutional neural network is trained to classify each image patch in the collected images. Quantitative evaluation conducted on a data set of 500 images of size 3264 χ 2448, collected by a low-cost smart phone, demonstrates that the learned deep features with the proposed deep learning framework provide superior crack detection performance when compared with features extracted with existing hand-craft methods.

Road crack detection using deep convolutional neural network

Computer vision-based techniques were developed to overcome the limitations of visual inspection by trained human resources and to detect structural damage in images remotely, but most methods detect only specific types of damage, such as concrete or steel cracks. To provide quasi real-time simultaneous detection of multiple types of damages, a Faster Region-based Convolutional Neural Network (Faster R-CNN)-based structural visual inspection method is proposed. To realize this, a database including 2,366 images (with 500 × 375 pixels) labeled for five types of damages—concrete crack, steel corrosion with two levels (medium and high), bolt corrosion, and steel delamination—is developed. Then, the architecture of the Faster R-CNN is modified, trained, validated, and tested using this database. Results show 90.6%, 83.4%, 82.1%, 98.1%, and 84.7% average precision (AP) ratings for the five damage types, respectively, with a mean AP of 87.8%. The robustness of the trained Faster R-CNN is evaluated and demonstrated using 11 new 6,000 × 4,000-pixel images taken of different structures. Its performance is also compared to that of the traditional CNN-based method. Considering that the proposed method provides a remarkably fast test speed (0.03 seconds per image with 500 × 375 resolution), a framework for quasi real-time damage detection on video using the trained networks is developed.

Autonomous Structural Visual Inspection Using Region-Based Deep Learning for Detecting Multiple Damage Types

Regular inspection of nuclear power plant components is important to guarantee safe operations. However, current practice is time consuming, tedious, and subjective, which involves human technicians reviewing the inspection videos and identifying cracks on reactors. A few vision-based crack detection approaches have been developed for metallic surfaces, and they typically perform poorly when used for analyzing nuclear inspection videos. Detecting these cracks is a challenging task since they are tiny, and noisy patterns exist on the components’ surfaces. This study proposes a deep learning framework, based on a convolutional neural network (CNN) and a Naive Bayes data fusion scheme, called NB-CNN, to analyze individual video frames for crack detection while a novel data fusion scheme is proposed to aggregate the information extracted from each video frame to enhance the overall performance and robustness of the system. To this end, a CNN is proposed to detect crack patches in each video frame, while the proposed data fusion scheme maintains the spatiotemporal coherence of cracks in videos, and the Naive Bayes decision making discards false positives effectively. The proposed framework achieves a 98.3% hit rate against 0.1 false positives per frame that is significantly higher than state-of-the-art approaches as presented in this paper.

NB-CNN: Deep Learning-Based Crack Detection Using Convolutional Neural Network and Naïve Bayes Data Fusion

The CrackNet, an efficient architecture based on the Convolutional Neural Network (CNN), is proposed in this article for automated pavement crack detection on 3D asphalt surfaces with explicit objective of pixel-perfect accuracy. Unlike the commonly used CNN, CrackNet does not have any pooling layers which downsize the outputs of previous layers. CrackNet fundamentally ensures pixel-perfect accuracy using the newly developed technique of invariant image width and height through all layers. CrackNet consists of five layers and includes more than one million parameters that are trained in the learning process. The input data of the CrackNet are feature maps generated by the feature extractor using the proposed line filters with various orientations, widths, and lengths. The output of CrackNet is the set of predicted class scores for all pixels. The hidden layers of CrackNet are convolutional layers and fully connected layers. CrackNet is trained with 1,800 3D pavement images and is then demonstrated to be successful in detecting cracks under various conditions using another set of 200 3D pavement images. The experiment using the 200 testing 3D images showed that CrackNet can achieve high Precision (90.13%), Recall (87.63%) and F-measure (88.86%) simultaneously. Compared with recently developed crack detection methods based on traditional machine learning and imaging algorithms, the CrackNet significantly outperforms the traditional approaches in terms of F-measure. Using parallel computing techniques, CrackNet is programmed to be efficiently used in conjunction with the data collection software.

Automated Pixel-Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep-Learning Network

Data has been acquired and used to train and test pavement management systems (PMS) and methods. The article discusses how PMS require detailed information on the current state of the roads in order to take appropriate actions to optimize expenditures on maintenance and rehabilitation. The presence of cracks is a crucial aspect to be considered. A solution based on an instrumented vehicle that is equipped with an imaging system, two Inertial Profilers, a Differential Global Positioning System (DGPS), and a webcam is presented. Information about the state of the road is acquired at normal road speed. A method that is based on the use of Gabor filters is used to detect longitudinal and transverse cracks. The methodologies used to create Gabor filter banks and the use of the filtered images as descriptors for subsequent classifiers are discussed in detail in the article. The article also evaluates three different methodologies for setting the threshold of the classifiers. Finally, an AdaBoost algorithm is used for selecting and combining the classifiers, which improves the results provided by a single classifier. The article discusses how suitable results have been obtained in comparison with other reference works.

Road Crack Detection Using Visual Features Extracted by Gabor Filters

Many projects concerning the protection, conservation, restoration, and dissemination of cultural heritage are being carried out around the world due to its growing interest as a driving force of socio-economic development. The existence of reliable, digital three-dimensional (3D) models that allow for the planning and management of these projects in a remote and decentralized way is currently a growing necessity. There are many software tools to perform the modeling and complete three-dimensional documentation of the intervened monuments. However, the Architecture, Engineering and Construction (AEC) sector has adopted the Building Information Modeling (BIM) standard over the last few decades due to the progress that has been made in its qualities and capabilities. The complex modeling of cultural heritage through commercial BIM software leads to the consideration of the concept of Heritage BIM (H-BIM), which pursues the modeling of architectural elements, according to artistic, historical, and constructive typologies. In addition, H-BIM is considered to be an emerging technology that enables us to understand, document, advertize, and virtually reconstruct the built heritage. This article is a review of the existing literature on H-BIM and its effective implementation in the cultural heritage sector, exploring the effectiveness and the usefulness of the different methodologies that were developed to model families of elements of interest.

A Review of Heritage Building Information Modeling (H-BIM)

Long-term assessment of a service robot in a hotel environment

The classification of the images taken during the measurement of an architectural asset is an essential task within the digital documentation of cultural heritage. A large number of images are usually handled, so their classification is a tedious task (and therefore prone to errors) and habitually consumes a lot of time. The availability of automatic techniques to facilitate these sorting tasks would improve an important part of the digital documentation process. In addition, a correct classification of the available images allows better management and more efficient searches through specific terms, thus helping in the tasks of studying and interpreting the heritage asset in question. The main objective of this article is the application of techniques based on deep learning for the classification of images of architectural heritage, specifically through the use of convolutional neural networks. For this, the utility of training these networks from scratch or only fine tuning pre-trained networks is evaluated. All this has been applied to classifying elements of interest in images of buildings with architectural heritage value. As no datasets of this type, suitable for network training, have been located, a new dataset has been created and made available to the public. Promising results have been obtained in terms of accuracy and it is considered that the application of these techniques can contribute significantly to the digital documentation of architectural heritage.

Classification of Architectural Heritage Images Using Deep Learning Techniques

This paper provides the results of various trial experiments in a hotel environment carried out using Sacarino, an interactive bellboy robot We analysed which aspects of the robot design and behaviour are relevant in terms of user engagement and comfort when interacting with our social robot The experiments carried out focused on the influence over the proxemics, duration and effectiveness of the interaction, taking into account three dichotomous factors related with the robot design and behaviour: robot embodiment (with/without robotic body), status of the robot (awake/asleep) and who starts communication (robot/user) Results show that users tend to maintain a personal distance when interacting with an embodied robot and that embodiment engages users in maintaining longer interactions On the other hand, including a greeting model in a robot is useful in terms of engaging users to maintain longer interactions, and that an active-looking robot is more attractive to the participants, producing longer interactions than in the case of a passive-looking robot A Bellboy, social robot interacting guests in a hotel is presentedSocial robot design should take care of target user age that influences use distanceA robotic body encourages remarkably HCI (with respect to common computers)A two-step salutation can attract user attention while avoiding intimidationMultimodal systems are highly recommended in real, noisy environment

Eduardo Zalama

Papers

Road Crack Detection Using Visual Features Extracted by Gabor Filters

A Review of Heritage Building Information Modeling (H-BIM)

Long-term assessment of a service robot in a hotel environment

Classification of Architectural Heritage Images Using Deep Learning Techniques

A bellboy robot