Probability Distributions.- Linear Models for Regression.- Linear Models for Classification.- Neural Networks.- Kernel Methods.- Sparse Kernel Machines.- Graphical Models.- Mixture Models and EM.- Approximate Inference.- Sampling Methods.- Continuous Latent Variables.- Sequential Data.- Combining Models.

Pattern Recognition and Machine Learning

“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

Studying animal behavior can reveal how animals make decisions based on what they sense in their environment, but measuring behavior can be difficult and time-consuming. Computer programs that measure and analyze animal movement have made these studies faster and easier to complete. These tools have also made more advanced behavioral experiments possible, which have yielded new insights about how the brain organizes behavior. Recently, scientists have started using new machine learning tools called deep neural networks to measure animal behavior. These tools learn to measure animal posture – the positions of an animal’s body parts in space – directly from real data, such as images or videos, without being explicitly programmed with instructions to perform the task. This allows deep learning algorithms to automatically track the locations of specific animal body parts in videos faster and more accurately than previous techniques. This ability to learn from images also removes the need to attach physical markers to animals, which may alter their natural behavior. Now, Graving et al. have created a new deep learning toolkit for measuring animal behavior that combines components from previous tools with the latest advances in computer science. Simple modifications to how the algorithms are trained can greatly improve their performance. For example, adding connections between layers, or ‘neurons’, in the deep neural network and training the algorithm to learn the full geometry of the body – by drawing lines between body parts – both enhance its accuracy. As a result of adding these changes, the new toolkit can measure an animal's pose from previously unseen images with high speed and accuracy, after being trained on just 100 examples. Graving et al. tested their model on videos of fruit flies, zebras and locusts, and found that, after training, it was able to accurately track the animals’ movements. The new toolkit has an easy-to-use software interface and is freely available for other scientists to use and build on. The new toolkit may help scientists in many fields including neuroscience and psychology, as well as other computer scientists. For example, companies like Google and Apple use similar algorithms to recognize gestures, so making those algorithms faster and more efficient may make them more suitable for mobile devices like smartphones or virtual-reality headsets. Other possible applications include diagnosing and tracking injuries, or movement-related diseases in humans and livestock.

/pdf/deepposekit-a-software-toolkit-for-fast-and-robust-animal-4m2lw825xe.pdf

DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning

Computational Neuroethology: A Call to Action.

Deep learning tools for the measurement of animal behavior in neuroscience.

Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in three-dimensional (3D) space. Deep neural networks can estimate two-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these 2D measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. Here, we present DeepFly3D, a software that infers the 3D pose of tethered, adult Drosophila using multiple camera images. DeepFly3D does not require manual calibration, uses pictorial structures to automatically detect and correct pose estimation errors, and uses active learning to iteratively improve performance. We demonstrate more accurate unsupervised behavioral embedding using 3D joint angles rather than commonly used 2D pose data. Thus, DeepFly3D enables the automated acquisition of Drosophila behavioral measurements at an unprecedented level of detail for a variety of biological applications.

DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila.

Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of pose—the positions of each appendage—in 3-dimensional (3D) space. Recent advances in computer vision and machine learning have made it possible to use deep neural networks to estimate 2-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. Here we present DeepFly3D, a computational pipeline for inferring the 3D pose of tethered, adult Drosophila using multiple camera images. First, we introduce an approach for multiple-camera calibration using the animal itself rather than the typical checkerboard or similar external apparatus. Second, we present an iterative approach that robustly infers 3D pose using graphical models and deep-learning based 2D predictions from multiple cameras. False predictions are rejected using an optimization scheme based on dynamic programming and belief propagation. To close the loop, the corrected poses are used to retrain the 2D pose deep estimation network and to improve 3D pose estimation. Finally, we provide a graphical user interface (GUI) and active learning policy for interacting with, annotating, and correcting 3D pose data. Emphasizing the importance of our tool, we demonstrate that unsupervised behavioral embedding of 3D joint angles yields more accurate behavioral maps than those generated with 2D pose data because the latter are highly perspective-dependent. We provide our DeepFly3D deep network and weights, training data, computational pipeline, and nearly one million images of tethered, behaving flies along with corresponding 3D joint positions1. These tools make it possible to acquire high-fidelity behavioral measurements at an unprecedented level of precision and resolution for a variety of biological applications.

https://www.biorxiv.org/content/biorxiv/early/2019/05/31/640375.full.pdf

DeepFly3D: A deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila

Our goal is to capture the pose of real animals using synthetic training examples, without using any manual supervision. Our focus is on neuroscience model organisms, to be able to study how neural circuits orchestrate behaviour. Human pose estimation attains remarkable accuracy when trained on real or simulated datasets consisting of millions of frames. However, for many applications simulated models are unrealistic and real training datasets with comprehensive annotations do not exist. We address this problem with a new sim2real domain transfer method. Our key contribution is the explicit and independent modeling of appearance, shape and pose in an unpaired image translation framework. Our model lets us train a pose estimator on the target domain by transferring readily available body keypoint locations from the source domain to generated target images. We compare our approach with existing domain transfer methods and demonstrate improved pose estimation accuracy on Drosophila melanogaster (fruit fly), Caenorhabditis elegans (worm) and Danio rerio (zebrafish), without requiring any manual annotation on the target domain and despite using simplistic off-the-shelf animal characters for simulation, or simple geometric shapes as models. Our new datasets, code and trained models will be published to support future computer vision and neuroscientific studies.

/pdf/deformation-aware-unpaired-image-translation-for-pose-1wp6rkkrgz.pdf

Deformation-Aware Unpaired Image Translation for Pose Estimation on Laboratory Animals

Recovering a person's height from a single image is important for virtual garment fitting, autonomous driving and surveillance. However, it is also very challenging without absolute scale information. Here, we examine the rarely addressed case, where camera parameters and scene geometry are all unknown. Under this circumstances, scale is inherently ambiguous, and height can only be inferred from those statistics that are intrinsic to human anatomy and can be estimated from images directly, such as articulated pose, bone-length proportions, and facial features. Our contribution is twofold. First, we create a new human-height dataset that is three magnitudes larger than existing ones, by mining explicit height labels and propagating them to additional images through face recognition and assignment consistency. Second, we test a wide range of machine learning models (linear, shallow, and deep models) to capture the relation between image content and human height. We also show that performance is predominantly limited by dataset size. Our central finding is that height can only be estimated with large uncertainty. The remaining high variance demonstrates that the geometrically motivated scale ambiguity persists into the age of deep learning, which has important implications for how to pose monocular reconstruction, such as 3D human pose estimation, in a scale invariant way.

/pdf/what-face-and-body-shapes-can-tell-us-about-height-4voqzpbswb.pdf

What Face and Body Shapes Can Tell Us About Height

Markerless three-dimensional (3D) pose estimation has become an indispensable tool for kinematic studies of laboratory animals. Most current methods recover 3D poses by multi-view triangulation of deep network-based two-dimensional (2D) pose estimates. However, triangulation requires multiple synchronized cameras and elaborate calibration protocols that hinder its widespread adoption in laboratory studies. Here we describe LiftPose3D, a deep network-based method that overcomes these barriers by reconstructing 3D poses from a single 2D camera view. We illustrate LiftPose3D’s versatility by applying it to multiple experimental systems using flies, mice, rats and macaques, and in circumstances where 3D triangulation is impractical or impossible. Our framework achieves accurate lifting for stereotypical and nonstereotypical behaviors from different camera angles. Thus, LiftPose3D permits high-quality 3D pose estimation in the absence of complex camera arrays and tedious calibration procedures and despite occluded body parts in freely behaving animals. LiftPose3D infers three-dimensional poses from two-dimensional data or from limited three-dimensional data. The approach is illustrated for videos of behaving Drosophila, mice, rats and macaques.

Semih Günel

Papers

DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila.

DeepFly3D: A deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila

Deformation-Aware Unpaired Image Translation for Pose Estimation on Laboratory Animals

What Face and Body Shapes Can Tell Us About Height

LiftPose3D, a deep learning-based approach for transforming two-dimensional to three-dimensional poses in laboratory animals.