Humans perceive the three-dimensional structure of the world with apparent ease. However, despite all of the recent advances in computer vision research, the dream of having a computer interpret an image at the same level as a two-year old remains elusive. Why is computer vision such a challenging problem and what is the current state of the art? Computer Vision: Algorithms and Applications explores the variety of techniques commonly used to analyze and interpret images. It also describes challenging real-world applications where vision is being successfully used, both for specialized applications such as medical imaging, and for fun, consumer-level tasks such as image editing and stitching, which students can apply to their own personal photos and videos. More than just a source of recipes, this exceptionally authoritative and comprehensive textbook/reference also takes a scientific approach to basic vision problems, formulating physical models of the imaging process before inverting them to produce descriptions of a scene. These problems are also analyzed using statistical models and solved using rigorous engineering techniques Topics and features: structured to support active curricula and project-oriented courses, with tips in the Introduction for using the book in a variety of customized courses; presents exercises at the end of each chapter with a heavy emphasis on testing algorithms and containing numerous suggestions for small mid-term projects; provides additional material and more detailed mathematical topics in the Appendices, which cover linear algebra, numerical techniques, and Bayesian estimation theory; suggests additional reading at the end of each chapter, including the latest research in each sub-field, in addition to a full Bibliography at the end of the book; supplies supplementary course material for students at the associated website, http://szeliski.org/Book/. Suitable for an upper-level undergraduate or graduate-level course in computer science or engineering, this textbook focuses on basic techniques that work under real-world conditions and encourages students to push their creative boundaries. Its design and exposition also make it eminently suitable as a unique reference to the fundamental techniques and current research literature in computer vision.

/pdf/computer-vision-algorithms-and-applications-25dn6wu83j.pdf

Computer Vision: Algorithms and Applications

http://cvssp-data.eps.surrey.ac.uk/Personal/AdrianHilton/Publications_files/survey_techrep.pdf

A survey of advances in vision-based human motion capture and analysis

A common trend in object recognition is to detect and leverage the use of sparse, informative feature points. The use of such features makes the problem more manageable while providing increased robustness to noise and pose variation. In this work we develop an extension of these ideas to the spatio-temporal case. For this purpose, we show that the direct 3D counterparts to commonly used 2D interest point detectors are inadequate, and we propose an alternative. Anchoring off of these interest points, we devise a recognition algorithm based on spatio-temporally windowed data. We present recognition results on a variety of datasets including both human and rodent behavior.

/pdf/behavior-recognition-via-sparse-spatio-temporal-features-4pkkmfitbb.pdf

Behavior recognition via sparse spatio-temporal features

This article develops methods for determining visually appealing motion transitions using linear blending. Motion transitions are segues between two sequences of animation, and are important components for generating compelling animation streams in virtual environments and computer games. Methods involving linear blending are studied because of their efficiency, computational speed, and widespread use. Two methods of transition specification are detailed, center-aligned and start-end transitions. First, we compute a set of optimal weights for an underlying cost metric used to determine the transition points. We then evaluate the optimally weighted cost metric for generalizability, appeal, and robustness through a cross-validation and user study. Next, we develop methods for computing visually appealing blend lengths for two broad categories of motion. We empirically evaluate these results through user studies. Finally, we assess the importance of these techniques by determining the minimum sensitivity of viewers to transition durations, the just noticeable difference, for both center-aligned and start-end specifications.

Synthesis and evaluation of linear motion transitions

One way that artists create compelling character animations is by manipulating details of a character's motion. This process is expensive and repetitive. We show that we can make such motion editing more efficient by generalizing the edits an animator makes on short sequences of motion to other sequences. Our method predicts frames for the motion using Gaussian process models of kinematics and dynamics. These estimates are combined with probabilistic inference. Our method can be used to propagate edits from examples to an entire sequence for an existing character, and it can also be used to map a motion from a control character to a very different target character. The technique shows good generalization. For example, we show that an estimator, learned from a few seconds of edited example animation using our methods, generalizes well enough to edit minutes of character animation in a high-quality fashion. Learning is interactive: An animator who wants to improve the output can provide small, correcting examples and the system will produce improved estimates of motion. We make this interactive learning process efficient and natural with a fast, full-body IK system with novel features. Finally, we present data from interviews with professional character animators that indicate that generalizing and propagating animator edits can save artists significant time and work.

Generalizing motion edits with Gaussian processes

We discuss a method for creating animations that allows the animator to sketch an animation by setting a small number of keyframes on a fraction of the possible degrees of freedom. Motion capture data is then used to enhance the animation. Detail is added to degrees of freedom that were keyframed, a process we call texturing. Degrees of freedom that were not keyframed are synthesized. The method takes advantage of the fact that joint motions of an articulated figure are often correlated, so that given an incomplete data set, the missing degrees of freedom can be predicted from those that are present.

http://ivizlab.sfu.ca/arya/Papers/ACM/SIGGRAPH-02/MotionCaptureAssistedAnimation.pdf

Motion capture assisted animation: texturing and synthesis

This paper demonstrates a new visual motion estimation technique that is able to recover high degree-of-freedom articulated human body configurations in complex video sequences. We introduce the use and integration of a mathematical technique, the product of exponential maps and twist motions, into a differential motion estimation. This results in solving simple linear systems, and enables us to recover robustly the kinematic degrees-of-freedom in noise and complex self occluded configurations. A new factorization technique lets us also recover the kinematic chain model itself. We are able to track several human walk cycles, several wallaby hop cycles, and two walk cycels of the famous movements of Eadweard Muybridge's motion studies from the last century. To the best of our knowledge, this is the first computer vision based system that is able to process such challenging footage.

/pdf/twist-based-acquisition-and-tracking-of-animal-and-human-16a5sil00r.pdf

Twist Based Acquisition and Tracking of Animal and Human Kinematics

We describe a method for synthesizing joint angle and translation data based on the information in motion capture data. The synthetic data is realistic not only in that it resembles the original training data, but in that it has random variations that are statistically similar to what one would find in repeated measurements of the motion. To achieve this result, the training data is broken into frequency bands using a wavelet decomposition, and the information in these bands is used to create the synthetic data one frequency band at a time. The method takes into account the fact that there are correlations among numerous features of the data. For example, a point characterized by a particular time and frequency band will depend upon points close to it in time in other frequency bands. Such correlations are modeled with a kernel-based representation of the joint probability distributions of the features. The data is synthesized by sampling from these densities and improving the results using a new iterative maximization technique. We have applied this technique to the synthesis of joint angle and translation data of a wallaby hopping on a treadmill. The synthetic data was used to animate characters that have limbs proportional to the wallaby.

Animating by multi-level sampling

https://febs.onlinelibrary.wiley.com/doi/pdf/10.1016/S0014-5793%2898%2900594-8

Single molecule biochemistry using optical tweezers

Motion capture data is useful to an animator because it captures the exact style of a particular individual’s movements and has a life-like quality. However, often the data is not exactly what the animator needs. We demonstrate a method for using motion capture data as a starting point for creating synthetic motion data that addresses this problem by allowing the animator to specify hard constraints such as where the feet should contact the floor. The method captures the style of the motion as well as the life-like quality. It begins with an analysis phase, in which the data is divided into features such as frequency bands and correlations among joint angles, and represented with multidimensional kernel-based probability distributions. These distributions are then sampled in a synthesis phase, and optimized to yield the final animation.

Katherine Pullen

Papers

Motion capture assisted animation: texturing and synthesis

Twist Based Acquisition and Tracking of Animal and Human Kinematics

Animating by multi-level sampling

Single molecule biochemistry using optical tweezers

Synthesis of Cyclic Motions with Texture