Towards an automatic evaluation of retrieval performance with large scale image collections - Archive ouverte HAL Access content directly
Conference Papers Year : 2015

Towards an automatic evaluation of retrieval performance with large scale image collections

(1) , (2) , (2) , (1) , (2)
1
2

Abstract

The public availability of large-scale multimedia collections, such as the Yahoo Flick Creative Commons (YFCC) dataset, facilitates the evaluation of image retrieval systems in real- istic conditions. However, due to their size, the creation of exhaustive ground truth would require huge annotation ef- fort, even for limited sets of queries. This paper investigates whether it is possible to estimate retrieval performance in absence of manually created ground truth data. Our hypoth- esis is that it is possible to leverage existing weak user anno- tations (tags) to automatically build ground truth data. To test this hypothesis, we implemented a large-scale retrieval pipeline based on two state-of-The-art image descriptors and two compressed versions of each. The top 50 results ob- tained with each con-guration are manually annotated to estimate their performance. Alternately, we produce an au- tomatic performance estimation based on pre-existing user tags. The automatic performance estimations exhibit strong positive correlation with the manual ones and the corre- sponding system rankings are found to be similar. Hence, we conclude that despite being incomplete and sometimes imprecise, weak user annotations can be leveraged to assess retrieval performance. As a by-product, we release state-of- the-art image features for YFCC and a reusable evaluation package to encourage its use in the community.
Not file

Dates and versions

cea-01841022 , version 1 (17-07-2018)

Identifiers

Cite

A. Popescu, E. Spyromitros-Xoufis, S. Papadopoulos, Hervé Le Borgne, I. Kompatsiaris. Towards an automatic evaluation of retrieval performance with large scale image collections. MMCommons '15 Proceedings of the 2015 Workshop on Community-Organized Multimodal Mining: Opportunities for Novel Solutions, Oct 2015, Unknown, Australia. pp.7-12, ⟨10.1145/2814815.2814819⟩. ⟨cea-01841022⟩
28 View
0 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More