Can I read CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation on EtoBox?

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation by Hao, Wangli; Zhang, Zhaoxiang; Guan, He is a scholarly article available to read on EtoBox.

What is CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation about?

Visual and audio modalities are two symbiotic modalities underlying videos, which contain both common and complementary information. If they can be mined and fused sufficiently, performances of related video tasks can be significantly enhanced. However, due to the environmental interference or sensor fault, sometimes, only one modality exists while the other is abandoned or missing. By recovering the missing modality from the existing one based on the common information shared between them and the prior information of the specific modality, great bonus will be gained for various vision tasks. In this paper, we propose a Cross-Modal Cycle Generative Adversarial Network (CMCGAN) to handle cross-modal visual-audio mutual generation. Specifically, CMCGAN is composed of four kinds of subnetworks: audio-to-visual, visual-to-audio, audio-to-audio and visual-to-visual subnetworks respectively, which are organized in a cycle architecture. CMCGAN has several remarkable advantages. Firstly, CMCGAN unifies visual-audio mutual generation into a common framework by a joint corresponding adversarial loss. Secondly, through introducing a latent vector with Gaussian distribution, CMCGAN can handle

Author

Hao, Wangli; Zhang, Zhaoxiang; Guan, He

Published

2017

Language

EN