Towards an automatic evaluation of retrieval performance with large scale image collections

Abstract : The public availability of large-scale multimedia collections, such as the Yahoo Flick Creative Commons (YFCC) dataset, facilitates the evaluation of image retrieval systems in real- istic conditions. However, due to their size, the creation of exhaustive ground truth would require huge annotation ef- fort, even for limited sets of queries. This paper investigates whether it is possible to estimate retrieval performance in absence of manually created ground truth data. Our hypoth- esis is that it is possible to leverage existing weak user anno- tations (tags) to automatically build ground truth data. To test this hypothesis, we implemented a large-scale retrieval pipeline based on two state-of-The-art image descriptors and two compressed versions of each. The top 50 results ob- tained with each con-guration are manually annotated to estimate their performance. Alternately, we produce an au- tomatic performance estimation based on pre-existing user tags. The automatic performance estimations exhibit strong positive correlation with the manual ones and the corre- sponding system rankings are found to be similar. Hence, we conclude that despite being incomplete and sometimes imprecise, weak user annotations can be leveraged to assess retrieval performance. As a by-product, we release state-of- the-art image features for YFCC and a reusable evaluation package to encourage its use in the community.
Complete list of metadatas

https://hal-cea.archives-ouvertes.fr/cea-01841022
Contributor : Léna Le Roy <>
Submitted on : Tuesday, July 17, 2018 - 7:58:46 AM
Last modification on : Wednesday, January 23, 2019 - 2:39:26 PM

Identifiers

Collections

Citation

A. Popescu, E. Spyromitros-Xoufis, S. Papadopoulos, H. Le Borgne, I. Kompatsiaris. Towards an automatic evaluation of retrieval performance with large scale image collections. MMCommons '15 Proceedings of the 2015 Workshop on Community-Organized Multimodal Mining: Opportunities for Novel Solutions, Oct 2015, Unknown, Australia. pp.7-12, ⟨10.1145/2814815.2814819⟩. ⟨cea-01841022⟩

Share

Metrics

Record views

44