2023
Aguilera, Aurora Cobo; Olmos, Pablo M; Artés-Rodríguez, Antonio; Pérez-Cruz, Fernando
Regularizing transformers with deep probabilistic layers Artículo de revista
En: Neural Networks, 2023, ISSN: 0893-6080.
Resumen | Enlaces | BibTeX | Etiquetas: Deep learning, Missing data, Natural language processing, Regularization, Transformers, Variational auto-encoder
@article{AGUILERA2023,
title = {Regularizing transformers with deep probabilistic layers},
author = {Aurora Cobo Aguilera and Pablo M Olmos and Antonio Art\'{e}s-Rodr\'{i}guez and Fernando P\'{e}rez-Cruz},
url = {https://www.sciencedirect.com/science/article/pii/S0893608023000448},
doi = {https://doi.org/10.1016/j.neunet.2023.01.032},
issn = {0893-6080},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Neural Networks},
abstract = {Language models (LM) have grown non-stop in the last decade, from sequence-to-sequence architectures to attention-based Transformers. However, regularization is not deeply studied in those structures. In this work, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer. We study its advantages regarding the depth where it is placed and prove its effectiveness in several scenarios. Experimental result demonstrates that the inclusion of deep generative models within Transformer-based architectures such as BERT, RoBERTa, or XLM-R can bring more versatile models, able to generalize better and achieve improved imputation score in tasks such as SST-2 and TREC or even impute missing/noisy words with richer text.},
keywords = {Deep learning, Missing data, Natural language processing, Regularization, Transformers, Variational auto-encoder},
pubstate = {published},
tppubtype = {article}
}
Language models (LM) have grown non-stop in the last decade, from sequence-to-sequence architectures to attention-based Transformers. However, regularization is not deeply studied in those structures. In this work, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer. We study its advantages regarding the depth where it is placed and prove its effectiveness in several scenarios. Experimental result demonstrates that the inclusion of deep generative models within Transformer-based architectures such as BERT, RoBERTa, or XLM-R can bring more versatile models, able to generalize better and achieve improved imputation score in tasks such as SST-2 and TREC or even impute missing/noisy words with richer text.
2014
Montoya-Martinez, Jair; Artés-Rodríguez, Antonio; Pontil, Massimiliano; Hansen, Lars Kai
A Regularized Matrix Factorization Approach to Induce Structured Sparse-Low Rank Solutions in the EEG Inverse Problem Artículo de revista
En: EURASIP Journal on Advances in Signal Processing, vol. 2014, no 1, pp. 97, 2014, ISSN: 1687-6180.
Resumen | Enlaces | BibTeX | Etiquetas: Low rank, Matrix factorization, Nonsmooth-nonconvex optimization, Regularization, Structured sparsity
@article{Montoya-Martinez2014b,
title = {A Regularized Matrix Factorization Approach to Induce Structured Sparse-Low Rank Solutions in the EEG Inverse Problem},
author = {Jair Montoya-Martinez and Antonio Art\'{e}s-Rodr\'{i}guez and Massimiliano Pontil and Lars Kai Hansen},
url = {http://www.tsc.uc3m.es/~antonio/papers/P48_2014_A Regularized Matrix Factorization Approach to Induce Structured Sparse-Low Rank Solutions in the EEG Inverse Problem.pdf
http://asp.eurasipjournals.com/content/2014/1/97/abstract},
issn = {1687-6180},
year = {2014},
date = {2014-01-01},
journal = {EURASIP Journal on Advances in Signal Processing},
volume = {2014},
number = {1},
pages = {97},
publisher = {Springer},
abstract = {We consider the estimation of the Brain Electrical Sources (BES) matrix from noisy Electroencephalographic (EEG) measurements, commonly named as the EEG inverse problem. We propose a new method to induce neurophysiological meaningful solutions, which takes into account the smoothness, structured sparsity and low rank of the BES matrix. The method is based on the factorization of the BES matrix as a product of a sparse coding matrix and a dense latent source matrix. The structured sparse-low rank structure is enforced by minimizing a regularized functional that includes the l21-norm of the coding matrix and the squared Frobenius norm of the latent source matrix. We develop an alternating optimization algorithm to solve the resulting nonsmooth-nonconvex minimization problem. We analyze the convergence of the optimization procedure, and we compare, under different synthetic scenarios, the performance of our method respect to the Group Lasso and Trace Norm regularizers when they are applied directly to the target matrix.},
keywords = {Low rank, Matrix factorization, Nonsmooth-nonconvex optimization, Regularization, Structured sparsity},
pubstate = {published},
tppubtype = {article}
}
We consider the estimation of the Brain Electrical Sources (BES) matrix from noisy Electroencephalographic (EEG) measurements, commonly named as the EEG inverse problem. We propose a new method to induce neurophysiological meaningful solutions, which takes into account the smoothness, structured sparsity and low rank of the BES matrix. The method is based on the factorization of the BES matrix as a product of a sparse coding matrix and a dense latent source matrix. The structured sparse-low rank structure is enforced by minimizing a regularized functional that includes the l21-norm of the coding matrix and the squared Frobenius norm of the latent source matrix. We develop an alternating optimization algorithm to solve the resulting nonsmooth-nonconvex minimization problem. We analyze the convergence of the optimization procedure, and we compare, under different synthetic scenarios, the performance of our method respect to the Group Lasso and Trace Norm regularizers when they are applied directly to the target matrix.