UC Berkeley

In this data explosion epoch, data-driven structural health monitoring (SHM) and rapid damage assessment after natural hazards have become of great interest in civil engineering. In this dissertation, the state-of-the-art machine learning (ML) and deep learning (DL) technologies are introduced and implemented for vision-based and vibration-based SHM applications, respectively. Specifically, the dissertation consists of five core parts, namely (1) motivation, background and basics of ML/DL-based data-driven SHM, (2) general topics of vision-based SHM, (3) advanced topics of vision-based SHM, (4) ML in vibration-based SHM, and (5) conclusions and extensions.

Using DL in vision-based SHM is a relatively new research direction in civil and structural engineering. As researchers begin to apply these concepts to the structural engineering domain, several critical issues need to be addressed: (1) the lack of uniform automated response and damage detection principles based on domain knowledge, (2) the lack of benchmark datasets with well-labeled large amounts of data, and (3) the lack of interpretation of the trained ML/DL models. To address the first two issues, an automated and hierarchical framework has been proposed and developed herein: the PHI-Net for the PEER (Pacific Earthquake Engineering Research Center) Hub Image-Net. This framework consists of eight basic benchmark detection tasks based on current domain knowledge and past reconnaissance experience, namely (1) scene level, (2) damage state, (3) concrete cover spalling condition (material loss), (4) material type, (5) collapse mode, (6) component type, (7) damage level, and (8) damage type. According to the PHI-Net framework, a large number of structural images was collected, preprocessed, and labeled to form an open-source online large-scale multi-attribute image dataset, namely, the PHI-Net dataset, which currently contains 36,413 images with multiple labels. Before conducting comprehensive numerical experiments on the PHI-Net dataset, a preliminary study was performed firstly on a smaller scale dataset for validating the effectiveness and applicability of the DL methods in vision-based SHM along with the transfer learning (TL) technique. In the study of the TL, instead of training a network from scratch, two different strategies, namely feature extractor and fine-tuning with pre-trained models, e.g., VGGNet (Visual Geometry Group) and Residual Network (ResNet), were pursued. A series of computer experiments were designed based on the properties of these two strategies to study the relative optimal model setting and scope of application. With a preliminary sense from the TL experiments, a comprehensive investigation on applying DL in general vision tasks in the PHI-Net was made through comparative experiments with various DL models and training strategies, i.e., TL and data augmentation (DA). Promising results were achieved and reported, which provide the reference for future DL applications. Moreover, two direct applications of the PHI-Net were further performed, namely image-based post-disaster assessment of the 1999 Chi-Chi earthquake and the 2018 PHI-Net Challenge, which revealed the great potential and contribution of the PHI-Net in vision-based SHM. To address the ``black box'' issue implying that the working principle of DL is hard to understand by humans, several state-of-the-art visual interpretation methods were applied in explaining the DL models in SHM problems. Beyond the general view over the basic (non-expert) structural vision tasks, i.e., spalling condition task, several key factors including learning procedure, learned features from different network depths, training techniques, and level of semantic abstraction were effectively explored.

As one advanced technology in DL, Generative Adversarial Network (GAN) was investigated to overcome the shortcomings of conventional DA, alleviate data deficit, and finally improve the model performance under certain conditions. Unlike normal natural objects, the distribution of structural images is much more complex, thus, Deep Convolutional GAN (DCGAN) along with Leaf-Bootstrapping (LB) training method was proposed to improve the quality of synthetic images. To effectively and quantitatively evaluate the quality of the synthetic images generated by DCGAN to complement human inspection, Self-Inception Score (SIS) and Generalization Ability (GA) were proposed. Four experimental cases were designed to explore the performance of LB based GAN under different situations with the above-mentioned metrics. To further explore the potentials of using synthetic images to improve the classifier performance, a well-designed experiment was conducted under the constraints of limited computational resources and low-data regime, where a union training strategy, namely synthetic data fine-tuning (SDF), was proposed. Moreover, to improve the efficiency of GAN-based augmentation method and also to take the imbalanced-label issue into account, a semi-supervised learning GAN pipeline along with the balanced batch sampling technique was proposed, and named Balanced Semi-supervised GAN (BSS-GAN). A series of complementary experiments with respect to the detection of the reinforced concrete cover spalling was designed and conducted, where comparisons were made among conventional training pipeline, directly synthetic data aggregation and BSS-GAN. The experimental results demonstrated the effectiveness and robustness of these proposed methods.

Besides the vision-based SHM, the effectiveness of ML in vibration-based SHM was also explored while combining ML with the time series (TS) approaches. Autoregressive (AR) series model, as one type of the TS methods, is useful for structural damage detection in the area of vibration-based SHM. However, due to several limitations (e.g. non-stationary conditions and subjective feature selection), it did not gain much attention during the past few years. With the boosting trends of ML technologies, this study combined TS modeling and ML classification together to automatically select the features and overcome several limitations. First, an automated end-to-end detection framework/algorithm was proposed, namely auto-regressive integrated moving average–machine learning (ARIMA-ML) with modules for smoothing-segmentation-normalization-differencing (SSND), auto-stationarity check, ARIMA feature extraction, and ML classification. In addition, an ensemble method using voting mechanism was adopted in the ML classification module. Based on shaking table tests of a steel building frame, floor acceleration data were collected and labeled according to experimental observations and records. Subsequently, three damage detection tasks were designed for: (1) general damage state, (2) local damage state, and (3) local damage pattern, where the influence of overlapping in the segmentation sliding windows was considered. In addition, to better understand the ML models, the importance of the features was also discussed along with the classification results. Moreover, the results of numerical experiments indicated the robustness and promising performance of the proposed algorithm in all considered tasks, which is expected to be useful for future damage assessment of instrumented structures by structural engineers during rapid reconnaissance efforts after natural hazard events, e.g., earthquakes.