We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

/pdf/going-deeper-with-convolutions-1yobw2o2ds.pdf

Going deeper with convolutions

The measurement of meaning

Computer and Robot Vision Vol. 1, by R.M. Haralick and Linda G. Shapiro, Addison-Wesley, 1992, ISBN 0-201-10887-1.

Computer and Robot Vision

We address the challenge of sentiment analysis from visual content. In contrast to existing methods which infer sentiment or emotion directly from visual low-level features, we propose a novel approach based on understanding of the visual concepts that are strongly related to sentiments. Our key contribution is two-fold: first, we present a method built upon psychological theories and web mining to automatically construct a large-scale Visual Sentiment Ontology (VSO) consisting of more than 3,000 Adjective Noun Pairs (ANP). Second, we propose SentiBank, a novel visual concept detector library that can be used to detect the presence of 1,200 ANPs in an image. The VSO and SentiBank are distinct from existing work and will open a gate towards various applications enabled by automatic sentiment analysis. Experiments on detecting sentiment of image tweets demonstrate significant improvement in detection accuracy when comparing the proposed SentiBank based predictors with the text-based approaches. The effort also leads to a large publicly available resource consisting of a visual sentiment ontology, a large detector library, and the training/testing benchmark for visual sentiment analysis.

/pdf/large-scale-visual-sentiment-ontology-and-detectors-using-1zmipwohfi.pdf

Large-scale visual sentiment ontology and detectors using adjective noun pairs

Towards understanding the effects of individual gamification elements on intrinsic motivation and performance

Images can affect people on an emotional level. Since the emotions that arise in the viewer of an image are highly subjective, they are rarely indexed. However there are situations when it would be helpful if images could be retrieved based on their emotional content. We investigate and develop methods to extract and combine low-level features that represent the emotional content of an image, and use these for image emotion classification. Specifically, we exploit theoretical and empirical concepts from psychology and art theory to extract image features that are specific to the domain of artworks with emotional expression. For testing and training, we use three data sets: the International Affective Picture System (IAPS); a set of artistic photography from a photo sharing site (to investigate whether the conscious use of colors and textures displayed by the artists improves the classification); and a set of peer rated abstract paintings to investigate the influence of the features and ratings on pictures without contextual content. Improved classification results are obtained on the International Affective Picture System (IAPS), compared to state of the art work.

Affective image classification using features inspired by psychology and art theory

In a smart home system, a camera-based fall detector at elderly homes leads to immediate alarming and helping. In this paper we propose an approach for the detection of falls based on multiple cameras. Based on semantic driven features, fall detection is done in 3D and fuzzy logic is used to estimate confidence values for different human postures as well as for the incidence of a fall/no fall. Emphasis is given on simplicity, low computational effort and fast processing. Therefore, based on an evaluation on 73 test sequences, we show the applicability of the method for videos with low spatial resolution and frame rate.

Detecting falls at homes using a network of low-resolution cameras

Diaries have transformed over the last decade. Originally in handwritten text format, photo albums and visual diaries became popular as photography became commonly available. Traditionally intended to remain private, the dimension of audience was added in blogs along with Internet communication. However, humans have a limited capacity to record their lives. Lifelogs overcome this limit and collect and store a person’s personal information digitally. This can be done by recording all computer and cell phone activity and mobile context (e.g. GPS), but also adding multiple wearable sensors such as “always-on” cameras or bio-sensors. The enormous amounts of data created needs to be processed in order to be made useful to humans. In this paper, we review state-of-the-art literature on wearable diaries and lifelogging systems, and discuss the key issues and main challenges. 1 Diaries, Blogs and Lifelogs Humans record their history, from global events that move the world to small everyday life events that they consider important. In this paper, we review the current trends of personal record keeping. We focus on approaches using wearable sensors to passively capture data. First, forms of personal record keeping are discussed, especially how diaries have changed over time, becoming increasingly multimodal. Section 2 describes key issues and main challenges of building modern diary systems. In Section 3 we present selected projects which offer unique diary solutions. We conclude with thoughts on future challenges are offered. 1.1. Traditional diary A diary is a sequence of entries arranged chronologically, created to report on what has happened over the course of a period of time. Personal diaries usually include the writer’s thoughts and feelings. Originally handwritten in the form of books, the diary transformed from paper to electronic formats. Along with new media, new forms of diaries also developed. 1.1.1 Visual diary Since photography became commonly available, photos were added to the text in the diary to illustrate events. In some cases the emphasis shifted towards photography altogether and it became common to create photo albums with short text comments. Photos simplified the record keeping and added a further dimension to the records that made the memories seem more real [26]. Sontag [26] goes even further and says that it is the photographic capture of reality that gives us the feeling of the realness of our lives, it helps us reconstruct our personal history. We don’t believe our perception until a photo confirms it [26] as is also illustrated by the popular Internet-phrase “pics or it didn’t happen” ∗This work was supported by FFG, project no. 830043 demanding photographic proof of an unbelievable story. Artists use visual diaries to sketch drawings of their ideas and to collect images or other media. They serve them as inspiration and as a means to reflect on their artistic growth [1]. In travel diaries the author might even add small souvenirs, brochures, postcards or other nostalgic items to the diary. 1.1.2 Blogs The most distinct and today probably also most popular form of diary are blogs. While the “paper version” was traditionally intended to remain private, this aspect widely changed as people adopted the new medium to chronicle their lives with the additional dimension of audience. Although the reasons given by bloggers such as documenting my life, expressing opinions, letting off steam (shouts, feelings and thoughts), inspiration (“to write is to think”), and building communities, the bloggers are conscious about their audience and censor the information they publish [21]. Admittedly, the entries serve rather for self-presentation and narcissism than for creating an extensive digital memory [21]. Still, the main point of a traditional diary (public or private) is that the content is actively generated by the user. However, documenting a life takes discipline and effort and, hence, the amount of information in the traditional diary is constrained either by the capacity of the medium or the writer. This limit forces the individual to filter the content he/she writes down. 1.2. LifeLog This issue of necessity (and difficulty) to limit information that can be preserved in a diary by a single human is being addressed by new technologies. Technological advancements today make it possible to explore new ways and possibilities to capture, collect and store information. First envisioned by Vannevar Bush in 1945 [5], a LifeLog presents the notion to capture and store a whole lifetime of a person’s personal information digitally, so that it can be retrieved whenever needed. In 2001, this idea was revived by an experiment of Gordon Bell [2], who scanned all of his paperwork, photographs, medical records and other personal data. The initial focus for lifelogging was on desktop applications only, however, it has shifted towards mobile access and capture. Mobile devices are capable of more than just storing they capture data passively, without the user’s conscious initiative. Lifelog systems collect a variety of signals subsets of the following data categories (as described in [7]): Passive visual capture: Wearable “always-on” cameras automatically take images or videos. Biometrics: Wearable sensors measuring bio-signals, such as heart rate, galvanic skin response, skin temperature or body motion. Mobile context: Cell phones can provide information such as location cues in the form of GPS data, wireless network presence and GSM location data. Co-present Bluetooth devices may indicate people present nearby. Mobile activity: Call logs, SMSs, even email-logs, activity on the web and social networks sites can be gathered from mobile phones. Desktop/laptop computer activity: Basically all PC/laptop activity of the user can be monitored, the time and duration of each task measured, documents saved, etc. Active capture: Indirect (writing blogs is monitored as computer activity) or direct (add photos, write comments and annotate the lifelog’s content) capture. Such technologies, however, are not limited to the obvious use of personal reminiscence only. Various applications featuring at least a subset of these possibilities have been used, e.g. in medical and therapeutic solutions, security enhancement, or to encourage self-reflection.

http://oagm2011.joanneum.at/papers/32.pdf

Affective computing for wearable diary and lifelogging systems: An overview

Visual lifelogging is a new concept of recording one’s daily life by using wearable camera to automatically capture images of one’s surroundings. Keyframe selection is a crucial process for summarizing lifelog image sequences. In the visual lifelog domain, images are passively captured in predefined time intervals, e.g. one image every 30 seconds. This results in variable visual quality in the image sequences. Contrary to videos, two consecutive frames are not necessarily similar in lifelog sequences. Thus, video processing techniques are not directly applicable. We propose a keyframe selection technique based on measuring image quality and distance to the middle frame. Based on the proposed evaluation framework, 81.6 % of keyframes selected by our approach are accepted, whereas only 70.4 % are accepted when using the middle frames as keyframes. Additionally, Ground Truth (GT) keyframes are investigated in terms of image quality and the position in time relative to their events. This provides information about their distributions and explain the results.

A Keyframe Selection of Lifelog Image Sequences

In the context of ambient assisted living, camera-based fall detectors at elderly homes leads to immediate alarming and helping. In this paper we propose a novel approach for the detection of falls based on multiple cameras. Based on semantic driven features fall detection is done in 2D and each camera decides on its own, if a fall has occurred. Fuzzy logic is both used to estimate confidence values for a fall/no fall in the single cameras as well as in the final voting step where the individual decisions are fused to an overall decision. Emphasis is given on simplicity, low computational effort and fast processing. We demonstrate the method and give results on 73 video sequences.

/pdf/fusion-of-data-from-multiple-cameras-for-fall-detection-lz5mliw0qy.pdf

Jana Machajdik

Papers

Affective image classification using features inspired by psychology and art theory

Detecting falls at homes using a network of low-resolution cameras

Affective computing for wearable diary and lifelogging systems: An overview

A Keyframe Selection of Lifelog Image Sequences

Fusion of Data from Multiple Cameras for Fall Detection