Robot imitation based on observations of the human movement is a challenging problem as the structure of the human demonstrator and the robot learner are usually different. A movement that can be demonstrated well by a human may not be kinematically feasible for robot reproduction. A common approach to solve this kinematic mapping is to retarget predefined corresponding parts of the human and the robot kinematic structure. When such a correspondence is not available, manual scaling of the movement amplitude and the positioning of the demonstration in relation to the reference frame of the robot may be required. This letter's contribution is a method that eliminates both the need of human-robot structural associations-and therefore is less sensitive to the type of robot kinematics-and searches for the optimal location and adaptation of the human demonstration, such that the robot can accurately execute the optimized solution. The method defines a cost that quantifies the quality of the kinematic mapping and decreases it in conjunction with task-specific costs such as via-points and obstacles. We demonstrate the method experimentally where a real golf swing recorded via marker tracking is generalized to different speeds on the embodiment of a 7 degree-of-freedom (DoF) arm. In simulation, we compare solutions of robots with different kinematic structures.

Acquiring and Generalizing the Embodiment Mapping From Human Observations to Robot Skills

The ability to identify objects or region correspondences between consecutive frames of a given hand-drawn animation sequence is an indispensable tool for automating animation modification tasks such as sequence-wide recoloring or shape-editing of a specific animated character. Existing correspondence identification methods heavily rely on appearance features, but these features alone are insufficient to reliably identify region correspondences when there exist occlusions or when two or more objects share similar appearances. To resolve the above problems, manual assistance is often required. In this paper, we propose a new correspondence identification method which considers both appearance features and motions of regions in a global manner. We formulate correspondence likelihoods between temporal region pairs as a network flow graph problem which can be solved by a well-established optimization algorithm. We have evaluated our method with various animation sequences and results show that our method consistently outperforms the state-of-the-art methods without any user guidance.

Globally optimal toon tracking

A new image registration algorithm using SDTR

DiLight: Digital light table – Inbetweening for 2D animations using guidelines

Automatic animation line art colorization is a challenging computer vision problem, since the information of the line art is highly sparse and abstracted and there exists a strict requirement for the color and style consistency between frames. Recently, a lot of Generative Adversarial Network (GAN) based image-to-image translation methods for single line art colorization have emerged. They can generate perceptually appealing results conditioned on line art images. However, these methods can not be adopted for the purpose of animation colorization because there is a lack of consideration of the in-between frame consistency. Existing methods simply input the previous colored frame as a reference to color the next line art, which will mislead the colorization due to the spatial misalignment of the previous colored frame and the next line art especially at positions where apparent changes happen. To address these challenges, we design a kind of correlation matching feature transfer model (called CMFT) to align the colored reference feature in a learnable way and integrate the model into an U-Net based generator in a coarse-to-fine manner This enables the generator to transfer the layer-wise synchronized features from the deep semantic code to the content progressively. Extension evaluation shows that CMFT model can effectively improve the in-between consistency and the quality of colored frames especially when the motion is intense and diverse.

/pdf/line-art-correlation-matching-feature-transfer-network-for-3vfj0kjj76.pdf

Line Art Correlation Matching Feature Transfer Network for Automatic Animation Colorization

Automatic cartoon matching in computer-assisted animation production

A redirection method of two-dimensional animation role actions relates to production of two-dimensional animations and includes that: a framework structure of a target animation role is defined on the basis of a source animation role and the target animation role which are similar in shape and on the basis of taking the source animation role and the target animation role as reference; the source animation role is subjected to framework extraction, a framework is subjected to feature description, scales and directions are distributed for framework data, and high-dimensional feature vectors are generated; a motion trail of the framework data in the source animation role is analyzed to obtain a framework motion trail of the source animation role; generation of a framework trail of the target animation role from the framework motion trail of the source animation role is achieved according to the obtained framework motion trail of the source animation role by combining a framework of the defined target animation role; and driving shape deformation of the framework is achieved according to the framework motion trail of the target animation role by combining the framework of the defined target animation role, and redirection of two-dimensional animation role actions is achieved

Zhijun Song

Papers

Automatic cartoon matching in computer-assisted animation production

Redirection method of two-dimensional animation role actions

Skeleton correspondence construction and its applications in animation style reusing