SUN Database: Exploring a Large Collection of Scene Categories

TLDR: The Scene Understanding database is proposed, a nearly exhaustive collection of scenes categorized at the same level of specificity as human discourse that contains 908 distinct scene categories and 131,072 images.

Abstract: Progress in scene understanding requires reasoning about the rich and diverse visual environments that make up our daily experience. To this end, we propose the Scene Understanding database, a nearly exhaustive collection of scenes categorized at the same level of specificity as human discourse. The database contains 908 distinct scene categories and 131,072 images. Given this data with both scene and object labels available, we perform in-depth analysis of co-occurrence statistics and the contextual relationship. To better understand this large scale taxonomy of scene categories, we perform two human experiments: we quantify human scene recognition accuracy, and we measure how typical each image is of its assigned scene category. Next, we perform computational experiments: scene recognition with global image features, indoor versus outdoor classification, and "scene detection," in which we relax the assumption that one image depicts only one scene category. Finally, we relate human experiments to machine performance and explore the relationship between human and machine recognition errors and the relationship between image "typicality" and machine recognition accuracy.

Figures

Fig. 22 For each feature, we plot the proportion of categories for which the largest incorrect (off-diagonal) confusion is the same category as the largest human confusion.

Fig. 7 Per-image object statistics in the SUN database.

Fig. 23 Selected SUN scene classification results using all features.

Fig. 1 Examples of scene categories in our SUN database.

Fig. 8 Average object scale compared to PASCAL VOC and ImageNet ILSVRC. SUN polygon is computed using the normalized area of the bounding polygon, and SUN box is computed using a the bounding box around the object. The blue text lists examples of object categories with very small areas in a typical image, with their average scale in per mil image pixels.

Fig. 14 Top row: SUN categories with the lowest human recognition rate. Below each of these categories, in the remaining three rows, are the most confusing classes for that category.