View in EDS

A symmetrical attention-assisted multi-modal fusion network under uncertain absent modalities.

Saved in:
Bibliographic Details
Title: A symmetrical attention-assisted multi-modal fusion network under uncertain absent modalities.
Authors: Li, Jiayao1,2 (AUTHOR) lijiayao@sxau.edu.cn, Cai, Saihua1,3 (AUTHOR) caisaih@ujs.edu.cn, Zhao, Kaiyi4 (AUTHOR) zhaokaiyi@nuc.edu.cn, Sun, Ruizhi1,2 (AUTHOR) sunruizhi@cau.edu.cn, Yuan, Gang2 (AUTHOR) yuangang@cau.edu.cn, Chen, Zeqiu2 (AUTHOR) chenzq@cau.edu.cn
Source: Expert Systems with Applications. May2026, Vol. 309, pN.PAG-N.PAG. 1p.
Subjects: Attention, Multisensor data fusion, Missing data (Statistics), Transformer models, Machine learning
Abstract: The research of absent modalities learning is an important challenge in the field of multi-modal learning. A series of methods have been proposed to address the degraded prediction performance due to missing modalities. However, existing studies ignore the deep symmetric information of different modalities and lack considering the uncertainty of missing modalities. To address these two challenges, we propose a S ymmetrical attention-assisted M ulti-modal F usion N etwork called SMFN under uncertain absent modalities. First, the modality representation network is utilized to represent different modalities in order to obtain the semantic features of different modalities; Then, the symmetrical attention-assisted module is designed for symmetrically exploring both intra- and inter-modal feature information for tackling the first challenge; Next, a multi-modal fusion module is introduced to map the feature information of own modality and different modalities into the same common space, thereby enhancing the robustness of uncertain missing modalities; Finally, the dependency between modalities is learned using the transformer to accomplish prediction is accomplished. In addition, a pre-training model is also designed to train the complete modalities for improving the prediction accuracy. Extensive experimental results on benchmark datasets demonstrate that compared to the state-of-the-art models, the SMFN improves the accuracy for 6.52% and the F1-score for 11.59% on average, it also exhibits good robustness and convergence. [ABSTRACT FROM AUTHOR]
Copyright of Expert Systems with Applications is the property of Pergamon Press - An Imprint of Elsevier Science and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Description
Abstract:The research of absent modalities learning is an important challenge in the field of multi-modal learning. A series of methods have been proposed to address the degraded prediction performance due to missing modalities. However, existing studies ignore the deep symmetric information of different modalities and lack considering the uncertainty of missing modalities. To address these two challenges, we propose a S ymmetrical attention-assisted M ulti-modal F usion N etwork called SMFN under uncertain absent modalities. First, the modality representation network is utilized to represent different modalities in order to obtain the semantic features of different modalities; Then, the symmetrical attention-assisted module is designed for symmetrically exploring both intra- and inter-modal feature information for tackling the first challenge; Next, a multi-modal fusion module is introduced to map the feature information of own modality and different modalities into the same common space, thereby enhancing the robustness of uncertain missing modalities; Finally, the dependency between modalities is learned using the transformer to accomplish prediction is accomplished. In addition, a pre-training model is also designed to train the complete modalities for improving the prediction accuracy. Extensive experimental results on benchmark datasets demonstrate that compared to the state-of-the-art models, the SMFN improves the accuracy for 6.52% and the F1-score for 11.59% on average, it also exhibits good robustness and convergence. [ABSTRACT FROM AUTHOR]
ISSN:09574174
DOI:10.1016/j.eswa.2026.131179