Service interruption on Monday 11 July from 12:30 to 13:00: all the sites of the CCSD (HAL, EpiSciences, SciencesConf, AureHAL) will be inaccessible (network hardware connection).
Skip to Main content Skip to Navigation
Conference papers

Towards an automatic evaluation of retrieval performance with large scale image collections

Abstract : The public availability of large-scale multimedia collections, such as the Yahoo Flick Creative Commons (YFCC) dataset, facilitates the evaluation of image retrieval systems in real- istic conditions. However, due to their size, the creation of exhaustive ground truth would require huge annotation ef- fort, even for limited sets of queries. This paper investigates whether it is possible to estimate retrieval performance in absence of manually created ground truth data. Our hypoth- esis is that it is possible to leverage existing weak user anno- tations (tags) to automatically build ground truth data. To test this hypothesis, we implemented a large-scale retrieval pipeline based on two state-of-The-art image descriptors and two compressed versions of each. The top 50 results ob- tained with each con-guration are manually annotated to estimate their performance. Alternately, we produce an au- tomatic performance estimation based on pre-existing user tags. The automatic performance estimations exhibit strong positive correlation with the manual ones and the corre- sponding system rankings are found to be similar. Hence, we conclude that despite being incomplete and sometimes imprecise, weak user annotations can be leveraged to assess retrieval performance. As a by-product, we release state-of- the-art image features for YFCC and a reusable evaluation package to encourage its use in the community.
Complete list of metadata
Contributor : Léna Le Roy Connect in order to contact the contributor
Submitted on : Tuesday, July 17, 2018 - 7:58:46 AM
Last modification on : Thursday, February 17, 2022 - 10:08:05 AM



A. Popescu, E. Spyromitros-Xoufis, S. Papadopoulos, Hervé Le Borgne, I. Kompatsiaris. Towards an automatic evaluation of retrieval performance with large scale image collections. MMCommons '15 Proceedings of the 2015 Workshop on Community-Organized Multimodal Mining: Opportunities for Novel Solutions, Oct 2015, Unknown, Australia. pp.7-12, ⟨10.1145/2814815.2814819⟩. ⟨cea-01841022⟩



Record views