2023
Moreno-Pino, Fernando; Olmos, Pablo M; Artés-Rodríguez, Antonio
Deep Autoregressive Models with Spectral Attention Artículo de revista
En: Pattern Recognition, pp. 109014, 2023, ISSN: 0031-3203.
Resumen | Enlaces | BibTeX | Etiquetas: Attention models, Deep learning, Filtering, global-local contexts, Signal processing, spectral domain attention, time series forecasting
@article{MORENOPINO2022109014,
title = {Deep Autoregressive Models with Spectral Attention},
author = {Fernando Moreno-Pino and Pablo M Olmos and Antonio Art\'{e}s-Rodr\'{i}guez},
url = {https://www.sciencedirect.com/science/article/pii/S0031320322004940},
doi = {https://doi.org/10.1016/j.patcog.2022.109014},
issn = {0031-3203},
year = {2023},
date = {2023-01-01},
urldate = {2022-01-01},
journal = {Pattern Recognition},
pages = {109014},
abstract = {Time series forecasting is an important problem across many domains, playing a crucial role in multiple real-world applications. In this paper, we propose a forecasting architecture that combines deep autoregressive models with a Spectral Attention (SA) module, which merges global and local frequency domain information in the model’s embedded space. By characterizing in the spectral domain the embedding of the time series as occurrences of a random process, our method can identify global trends and seasonality patterns. Two spectral attention models, global and local to the time series, integrate this information within the forecast and perform spectral filtering to remove time series’s noise. The proposed architecture has a number of useful properties: it can be effectively incorporated into well-known forecast architectures, requiring a low number of parameters and producing explainable results that improve forecasting accuracy. We test the Spectral Attention Autoregressive Model (SAAM) on several well-known forecast datasets, consistently demonstrating that our model compares favorably to state-of-the-art approaches.},
keywords = {Attention models, Deep learning, Filtering, global-local contexts, Signal processing, spectral domain attention, time series forecasting},
pubstate = {published},
tppubtype = {article}
}
Time series forecasting is an important problem across many domains, playing a crucial role in multiple real-world applications. In this paper, we propose a forecasting architecture that combines deep autoregressive models with a Spectral Attention (SA) module, which merges global and local frequency domain information in the model’s embedded space. By characterizing in the spectral domain the embedding of the time series as occurrences of a random process, our method can identify global trends and seasonality patterns. Two spectral attention models, global and local to the time series, integrate this information within the forecast and perform spectral filtering to remove time series’s noise. The proposed architecture has a number of useful properties: it can be effectively incorporated into well-known forecast architectures, requiring a low number of parameters and producing explainable results that improve forecasting accuracy. We test the Spectral Attention Autoregressive Model (SAAM) on several well-known forecast datasets, consistently demonstrating that our model compares favorably to state-of-the-art approaches.
Aguilera, Aurora Cobo; Olmos, Pablo M; Artés-Rodríguez, Antonio; Pérez-Cruz, Fernando
Regularizing transformers with deep probabilistic layers Artículo de revista
En: Neural Networks, 2023, ISSN: 0893-6080.
Resumen | Enlaces | BibTeX | Etiquetas: Deep learning, Missing data, Natural language processing, Regularization, Transformers, Variational auto-encoder
@article{AGUILERA2023,
title = {Regularizing transformers with deep probabilistic layers},
author = {Aurora Cobo Aguilera and Pablo M Olmos and Antonio Art\'{e}s-Rodr\'{i}guez and Fernando P\'{e}rez-Cruz},
url = {https://www.sciencedirect.com/science/article/pii/S0893608023000448},
doi = {https://doi.org/10.1016/j.neunet.2023.01.032},
issn = {0893-6080},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Neural Networks},
abstract = {Language models (LM) have grown non-stop in the last decade, from sequence-to-sequence architectures to attention-based Transformers. However, regularization is not deeply studied in those structures. In this work, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer. We study its advantages regarding the depth where it is placed and prove its effectiveness in several scenarios. Experimental result demonstrates that the inclusion of deep generative models within Transformer-based architectures such as BERT, RoBERTa, or XLM-R can bring more versatile models, able to generalize better and achieve improved imputation score in tasks such as SST-2 and TREC or even impute missing/noisy words with richer text.},
keywords = {Deep learning, Missing data, Natural language processing, Regularization, Transformers, Variational auto-encoder},
pubstate = {published},
tppubtype = {article}
}
Language models (LM) have grown non-stop in the last decade, from sequence-to-sequence architectures to attention-based Transformers. However, regularization is not deeply studied in those structures. In this work, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer. We study its advantages regarding the depth where it is placed and prove its effectiveness in several scenarios. Experimental result demonstrates that the inclusion of deep generative models within Transformer-based architectures such as BERT, RoBERTa, or XLM-R can bring more versatile models, able to generalize better and achieve improved imputation score in tasks such as SST-2 and TREC or even impute missing/noisy words with richer text.
Galdo, Antía López; Guerrero-López, Alejandro; Olmos, Pablo M.; García, María Jesús Gómez
Detecting train driveshaft damages using accelerometer signals and Differential Convolutional Neural Networks Artículo de revista
En: Engineering Applications of Artificial Intelligence, vol. 126, pp. 106840, 2023, ISSN: 0952-1976.
Resumen | Enlaces | BibTeX | Etiquetas: Condition monitoring, Convolutional Neural Networks, Crack detection, Deep learning, Railway axles, Vibration signal
@article{LOPEZGALDO2023106840,
title = {Detecting train driveshaft damages using accelerometer signals and Differential Convolutional Neural Networks},
author = {Ant\'{i}a L\'{o}pez Galdo and Alejandro Guerrero-L\'{o}pez and Pablo M. Olmos and Mar\'{i}a Jes\'{u}s G\'{o}mez Garc\'{i}a},
url = {https://www.sciencedirect.com/science/article/pii/S0952197623010242},
doi = {https://doi.org/10.1016/j.engappai.2023.106840},
issn = {0952-1976},
year = {2023},
date = {2023-01-01},
journal = {Engineering Applications of Artificial Intelligence},
volume = {126},
pages = {106840},
abstract = {Maintaining railway axles is crucial to prevent catastrophic failures and enormous human and economic costs. In recent years, there has been a growing interest in the railway industry to adopt condition monitoring techniques to enhance the safety and efficiency of the rail transport system, which maintenance is currently based on periodic inspections. In this context, this work presents a technique for real-time crack diagnosis on railway axles, based on advanced 2D-Convolutional Neural Network (CNN) architectures applied to time\textendashfrequency representations of vibration signals. One of the critical novelties is introducing a differential CNN structure that captures the system’s statistical properties, enabling generalisation between different mechanical sets and conditions. The proposed system has been trained with data corresponding to a unique wheelset assembly, showing that the model is able to diagnose cracks on the three different wheelset tested in operation under 32 different combinations of conditions, such as load, speed, sense of rotation and vibration direction. Four different crack levels have been introduced, representing the maximum one a 0.08% of the axle diameter, and the method proposed achieved Area Under the Curve (AUC) score of 0.85, significantly outperforming results obtained with other architectures proposed in the state-of-the-art, the score of the next below is 0.76. The results demonstrate the effectiveness and practicality of this approach to accurately classify the four crack levels tested within a condition monitoring system for non-stationary conditions, that would enable reliable real-time diagnosis, thus paving the way towards a more robust and efficient railway axle maintenance system.},
keywords = {Condition monitoring, Convolutional Neural Networks, Crack detection, Deep learning, Railway axles, Vibration signal},
pubstate = {published},
tppubtype = {article}
}
Maintaining railway axles is crucial to prevent catastrophic failures and enormous human and economic costs. In recent years, there has been a growing interest in the railway industry to adopt condition monitoring techniques to enhance the safety and efficiency of the rail transport system, which maintenance is currently based on periodic inspections. In this context, this work presents a technique for real-time crack diagnosis on railway axles, based on advanced 2D-Convolutional Neural Network (CNN) architectures applied to time–frequency representations of vibration signals. One of the critical novelties is introducing a differential CNN structure that captures the system’s statistical properties, enabling generalisation between different mechanical sets and conditions. The proposed system has been trained with data corresponding to a unique wheelset assembly, showing that the model is able to diagnose cracks on the three different wheelset tested in operation under 32 different combinations of conditions, such as load, speed, sense of rotation and vibration direction. Four different crack levels have been introduced, representing the maximum one a 0.08% of the axle diameter, and the method proposed achieved Area Under the Curve (AUC) score of 0.85, significantly outperforming results obtained with other architectures proposed in the state-of-the-art, the score of the next below is 0.76. The results demonstrate the effectiveness and practicality of this approach to accurately classify the four crack levels tested within a condition monitoring system for non-stationary conditions, that would enable reliable real-time diagnosis, thus paving the way towards a more robust and efficient railway axle maintenance system.
2022
Pantoja-Rosero, B. G.; Achanta, R.; Kozinski, M.; Fua, P.; Perez-Cruz, Fernando; Beyer, K.
Generating LOD3 building models from structure-from-motion and semantic segmentation Artículo de revista
En: Automation in Construction, vol. 141, pp. 104430, 2022, ISSN: 0926-5805.
Resumen | Enlaces | BibTeX | Etiquetas: 3D building models, Deep learning, Digital twin, LOD models, Masonry buildings, Structure from motion
@article{PANTOJAROSERO2022104430,
title = {Generating LOD3 building models from structure-from-motion and semantic segmentation},
author = {B. G. Pantoja-Rosero and R. Achanta and M. Kozinski and P. Fua and Fernando Perez-Cruz and K. Beyer},
url = {https://www.sciencedirect.com/science/article/pii/S092658052200303X},
doi = {https://doi.org/10.1016/j.autcon.2022.104430},
issn = {0926-5805},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Automation in Construction},
volume = {141},
pages = {104430},
abstract = {This paper describes a pipeline for automatically generating level of detail (LOD) models (digital twins), specifically LOD2 and LOD3, from free-standing buildings. Our approach combines structure from motion (SfM) with deep-learning-based segmentation techniques. Given multiple-view images of a building, we compute a three-dimensional (3D) planar abstraction (LOD2 model) of its point cloud using SfM techniques. To obtain LOD3 models, we use deep learning to perform semantic segmentation of the openings in the two-dimensional (2D) images. Unlike existing approaches, we do not rely on complex input, pre-defined 3D shapes or manual intervention. To demonstrate the robustness of our method, we show that it can generate 3D building shapes from a collection of building images with no further input. For evaluating reconstructions, we also propose two novel metrics. The first is a Euclidean\textendashdistance-based correlation of the 3D building model with the point cloud. The second involves re-projecting 3D model facades onto source photos to determine dice scores with respect to the ground-truth masks. Finally, we make the code, the image datasets, SfM outputs, and digital twins reported in this work publicly available in github.com/eesd-epfl/LOD3_buildings and doi.org/10.5281/zenodo.6651663. With this work we aim to contribute research in applications such as construction management, city planning, and mechanical analysis, among others.},
keywords = {3D building models, Deep learning, Digital twin, LOD models, Masonry buildings, Structure from motion},
pubstate = {published},
tppubtype = {article}
}
This paper describes a pipeline for automatically generating level of detail (LOD) models (digital twins), specifically LOD2 and LOD3, from free-standing buildings. Our approach combines structure from motion (SfM) with deep-learning-based segmentation techniques. Given multiple-view images of a building, we compute a three-dimensional (3D) planar abstraction (LOD2 model) of its point cloud using SfM techniques. To obtain LOD3 models, we use deep learning to perform semantic segmentation of the openings in the two-dimensional (2D) images. Unlike existing approaches, we do not rely on complex input, pre-defined 3D shapes or manual intervention. To demonstrate the robustness of our method, we show that it can generate 3D building shapes from a collection of building images with no further input. For evaluating reconstructions, we also propose two novel metrics. The first is a Euclidean–distance-based correlation of the 3D building model with the point cloud. The second involves re-projecting 3D model facades onto source photos to determine dice scores with respect to the ground-truth masks. Finally, we make the code, the image datasets, SfM outputs, and digital twins reported in this work publicly available in github.com/eesd-epfl/LOD3_buildings and doi.org/10.5281/zenodo.6651663. With this work we aim to contribute research in applications such as construction management, city planning, and mechanical analysis, among others.
Pantoja-Rosero, B. G.; Oner, D.; Kozinski, M.; Achanta, R.; Fua, P.; Perez-Cruz, Fernando; Beyer, K.
TOPO-Loss for continuity-preserving crack detection using deep learning Artículo de revista
En: Construction and Building Materials, vol. 344, pp. 128264, 2022, ISSN: 0950-0618.
Resumen | Enlaces | BibTeX | Etiquetas: Crack detection, Deep learning, Masonry buildings, Post-earthquake assessment
@article{PANTOJAROSERO2022128264,
title = {TOPO-Loss for continuity-preserving crack detection using deep learning},
author = {B. G. Pantoja-Rosero and D. Oner and M. Kozinski and R. Achanta and P. Fua and Fernando Perez-Cruz and K. Beyer},
url = {https://www.sciencedirect.com/science/article/pii/S0950061822019250},
doi = {https://doi.org/10.1016/j.conbuildmat.2022.128264},
issn = {0950-0618},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Construction and Building Materials},
volume = {344},
pages = {128264},
abstract = {We present a method for segmenting cracks in images of masonry buildings damaged by earthquakes. Existing methods of crack detection fail to preserve the continuity of cracks, and their performance deteriorates with imprecise training labels. We address these problems by adapting an approach previously proposed for reconstructing roads in aerial images, in which a Convolutional Neural Network is trained with a loss function specifically designed to encourage the continuity of thin structures and to accommodate imprecise annotations. We evaluate combinations of three loss functions (the Mean Squared Error, the Dice loss and the new connectivity-oriented loss) on two datasets using TernausNet, a deep network shown to attain state-of-the-art accuracy in crack detection. We herein show that combining these three losses significantly improves the topology of the predictions quantitatively and qualitatively. We also propose a new continuity metric, named Cracks Per Patch (CPP), and share a new dataset of images of earthquake-affected urban scenes accompanied by crack annotations. The dataset and implementations are publicly available for future studies and benchmarking (https://github.com/eesd-epfl/topo_crack_detection and https://doi.org/10.5281/zenodo.6769028).},
keywords = {Crack detection, Deep learning, Masonry buildings, Post-earthquake assessment},
pubstate = {published},
tppubtype = {article}
}
We present a method for segmenting cracks in images of masonry buildings damaged by earthquakes. Existing methods of crack detection fail to preserve the continuity of cracks, and their performance deteriorates with imprecise training labels. We address these problems by adapting an approach previously proposed for reconstructing roads in aerial images, in which a Convolutional Neural Network is trained with a loss function specifically designed to encourage the continuity of thin structures and to accommodate imprecise annotations. We evaluate combinations of three loss functions (the Mean Squared Error, the Dice loss and the new connectivity-oriented loss) on two datasets using TernausNet, a deep network shown to attain state-of-the-art accuracy in crack detection. We herein show that combining these three losses significantly improves the topology of the predictions quantitatively and qualitatively. We also propose a new continuity metric, named Cracks Per Patch (CPP), and share a new dataset of images of earthquake-affected urban scenes accompanied by crack annotations. The dataset and implementations are publicly available for future studies and benchmarking (https://github.com/eesd-epfl/topo_crack_detection and https://doi.org/10.5281/zenodo.6769028).
2019
Peis, Ignacio; Olmos, Pablo M; Vera-Varela, Constanza; Barrigón, María Luisa; Courtet, Philippe; Baca-García, Enrique; Artes-Rodríguez, Antonio
Deep Sequential Models for Suicidal Ideation From Multiple Source Data Artículo de revista
En: IEEE Journal of Biomedical and Health Informatics, vol. 23, no 6, pp. 2286 - 2293, 2019.
Enlaces | BibTeX | Etiquetas: attention, Deep learning, EMA, RNN, Suicide
@article{AArtes19,
title = {Deep Sequential Models for Suicidal Ideation From Multiple Source Data},
author = {Ignacio Peis and Pablo M Olmos and Constanza Vera-Varela and Mar\'{i}a Luisa Barrig\'{o}n and Philippe Courtet and Enrique Baca-Garc\'{i}a and Antonio Artes-Rodr\'{i}guez},
doi = {10.1109/JBHI.2019.2919270},
year = {2019},
date = {2019-05-27},
journal = {IEEE Journal of Biomedical and Health Informatics},
volume = {23},
number = {6},
pages = {2286 - 2293},
keywords = {attention, Deep learning, EMA, RNN, Suicide},
pubstate = {published},
tppubtype = {article}
}