This paper surveys theoretical and practical issues associated with a particular type of information retrieval problem, namely that where the information need is pictorial. The paper is contextualised by the notion of a visually stimulated society, in which the ease of record creation and transmission in the visual medium is contrasted with the difficulty of gaining effective subject access to the world's stores of such records. The technological developments which, in casting the visual image in electronic form, have contributed so significantly to its availability are reviewed briefly, as a prelude to the main thrust of the paper. Concentrating on still and moving pictorial forms of the visual image, the paper dwells on issues related to the subject indexing of pictorial material and discusses four models of pictorial information retrieval corresponding with permutations of the verbal and visual modes for the representation of picture content and of information need.

Progress in documentation pictorial information retrieval

With the current rapid expansion in imaging technologies, access to collections of images is a subject of major interest. This exploratory research investigated image attributes typically noted by participants in a series of describing tasks involving activities such as viewing images, describing them for a retrieval system, and describing them from memory. Content analysis and descriptive statistics were used to characterize textual statements generated by participants; this analysis produced forty-seven image attributes which were grouped conceptually into twelve higher level classes of attributes. The data suggest that access to a wide range of attributes is needed to address all facets of interest to those using pictorial images. They further suggest that certain classes of attributes may appear more frequently in a set of tasks relating to the description of images, including literal objects, the human form and associated attributes, and color and location terms, More unexpectedly, terms describing the ‘story’ within the image also appeared frequently in this research. The disjunction between these results and those attributes typically addressed in traditional image indexing systems suggest revisiting assumptions upon which image indexing and retrieval systems are being created.

Attributes of images in describing tasks

In order to evaluate automated image annotation and object recognition algorithms, ground truth in the form of a set of images correctly annotated with text describing each image is required. In this paper, three image annotation approaches are reviewed: free text annotation, keyword annotation and annotation based on ontologies. The practical aspects of image annotation are then considered. We discuss the creation of keyword vocabularies for use in automated image annotation evaluation. As direct manual annotation of images requires much time and effort, we also review various methods to make the creation of ground truth more efficient. An overview of annotated image datasets for computer vision research is provided.

A survey of methods for image annotation

Vast amounts of artistic data is scattered on-line from both museums and art applications. Collecting, processing and studying it with respect to all accompanying attributes is an expensive process. With a motivation to speed up and improve the quality of categorical analysis in the artistic domain, in this paper we propose an efficient and accurate method for multi-task learning with a shared representation applied in the artistic domain. We continue to show how different multi-task configurations of our method behave on artistic data and outperform handcrafted feature approaches as well as convolutional neural networks. In addition to the method and analysis, we propose a challenge like nature to the new aggregated data set with almost half a million samples and structured meta-data to encourage further research and societal engagement.

OmniArt: Multi-task Deep Learning for Artistic Data Analysis.

Baselines are the starting point of any quantitative multimedia research, and benchmarks are essential for pushing those baselines further. In this article, we present baselines for the artistic domain with a new benchmark dataset featuring over 2 million images with rich structured metadata dubbed OmniArt. OmniArt contains annotations for dozens of attribute types and features semantic context information through concepts, IconClass labels, color information, and (limited) object-level bounding boxes. For our dataset we establish and present baseline scores on multiple tasks such as artist attribution, creation period estimation, type, style, and school prediction. In addition to our metadata related experiments, we explore the color spaces of art through different types and evaluate a transfer learning object recognition pipeline.

OmniArt: A Large-scale Artistic Benchmark

Iconclass is a classification of subjects, themes and motifs in Western art. It was developed and published in Holland, at the University of Leiden, with financial aid from both the Netherlands Organisation for the Advancement of Pure Research (Z. W.O.) and the Royal Netherlands Academy of Arts and Sciences. Iconclass is used in a growing number of projects, whether computerised or not. One of these projects could be the Witt Library pilot project, discussed elsewhere in this issue of the Art Libraries Journal (pp. 27-31). In this paper the classification as such is briefly explained, mainly with the help of two examples: scenes from the stories of St. Antony Abbot and Hercules. Information on the alphabetical index to the classification is given, mainly because this extensive part of Iconclass is something more than the usual index to a publication. Applications of the classification are dealt with in diagrammatic form; one application, the Iconclass bibliography, is discussed in greater detail.

L.D. Couprie

Papers

Iconclass: an iconographic classification system