Cross-modal classification by completing unimodal representations

T.Q.N. Tran 1, 2 H. Le Borgne 1 M. Crucianu 2
1 DIASI - Département Intelligence Ambiante et Systèmes Interactifs
2 CEDRIC - VERTIGO - CEDRIC. Bases de données avancées
CEDRIC - Centre d'études et de recherche en informatique et communications
Abstract : We argue that cross-modal classification, where models are trained on data from one modality (e.g. text) and applied to data from another (e.g. image), is a relevant problem in multimedia retrieval. We propose a method that addresses this specific problem, related to but different from cross-modal retrieval and bimodal classification. This method relies on a common latent space where both modalities have comparable representations and on an auxiliary dataset from which we build a more complete bimodal representation of any unimodal data. Evaluations on Pascal VOC07 and NUS-WIDE show that the novel representation method significantly improves the results compared to the use of a latent space alone. The level of performance achieved makes cross-modal classification a convincing choice for real applications.
Keywords : nocv2 Real applications Representation method Specific problems Unimodal Text processing Multimedia Retrieval Cross-modal
Document type :
Conference papers
Complete list of metadatas

Cited literature [24 references]  Display  Hide  Download
https://hal-cea.archives-ouvertes.fr/cea-01840417
Contributor : Léna Le Roy <>
Submitted on : Friday, January 10, 2020 - 4:29:25 PM
Last modification on : Monday, February 10, 2020 - 6:13:48 PM

File

ivlm08-tranA.pdf
Files produced by the author(s)

Identifiers

Collections

DRT | UNIV-HESAM | UNIV-PARIS-SACLAY | LIST | CNAM | CEA | CEA-UPSAY | CEDRIC-CNAM

Citation

T.Q.N. Tran, H. Le Borgne, M. Crucianu. Cross-modal classification by completing unimodal representations. iV&L-MM '16 Proceedings of the 2016 ACM workshop on Vision and Language Integration Meets Multimedia Fusion, Oct 2016, Amsterdam, Netherlands. pp.17-25, ⟨10.1145/2983563.2983570⟩. ⟨cea-01840417⟩

Export

BibTeX TEI DC DCterms EndNote

Share

Metrics

Record views

111

Files downloads

23

Contact

Informations

CCSD