scispace - formally typeset
Search or ask a question

Showing papers in "IEEE Transactions on Cognitive and Developmental Systems in 2019"


Journal ArticleDOI
TL;DR: It is proposed to apply domain adaptation to reduce the intersubject variance as well as technical discrepancies between datasets, and then train a subject-independent classifier on one dataset and test on the other, and show that using domain adaptation technique in a transductive adaptation setting can improve the accuracy significantly.
Abstract: Affective brain–computer interface (aBCI) introduces personal affective factors to human–computer interaction. The state-of-the-art aBCI tailors its classifier to each individual user to achieve accurate emotion classification. A subject-independent classifier that is trained on pooled data from multiple subjects generally leads to inferior accuracy, due to the fact that electroencephalography patterns vary from subject to subject. Transfer learning or domain adaptation techniques have been leveraged to tackle this problem. Existing studies have reported successful applications of domain adaptation techniques on SEED dataset. However, little is known about the effectiveness of the domain adaptation techniques on other affective datasets or in a cross-dataset application. In this paper, we focus on a comparative study on several state-of-the-art domain adaptation techniques on two datasets: 1) DEAP and 2) SEED. We demonstrate that domain adaptation techniques can improve the classification accuracy on both datasets, but not so effective on DEAP as on SEED. Then, we explore the efficacy of domain adaptation in a cross-dataset setting when the data are collected under different environments using different devices and experimental protocols. Here, we propose to apply domain adaptation to reduce the intersubject variance as well as technical discrepancies between datasets, and then train a subject-independent classifier on one dataset and test on the other. Experiment results show that using domain adaptation technique in a transductive adaptation setting can improve the accuracy significantly by 7.25%–13.40% compared to the baseline accuracy where no domain adaptation technique is used.

220 citations


Journal ArticleDOI
TL;DR: A deep SpiCNN, consisting of two convolutional layers trained using the unsupervised Convolutional STDP learning methodology, achieved classification accuracies of 91.1% and 97.6%, respectively, for inferring handwritten digits from the MNIST data set and a subset of natural images from the Caltech data set.
Abstract: Spiking neural networks (SNNs) have emerged as a promising brain inspired neuromorphic-computing paradigm for cognitive system design due to their inherent event-driven processing capability. The fully connected (FC) shallow SNNs typically used for pattern recognition require large number of trainable parameters to achieve competitive classification accuracy. In this paper, we propose a deep spiking convolutional neural network (SpiCNN) composed of a hierarchy of stacked convolutional layers followed by a spatial-pooling layer and a final FC layer. The network is populated with biologically plausible leaky-integrate-and-fire (LIF) neurons interconnected by shared synaptic weight kernels. We train convolutional kernels layer-by-layer in an unsupervised manner using spike-timing-dependent plasticity (STDP) that enables them to self-learn characteristic features making up the input patterns. In order to further improve the feature learning efficiency, we propose using smaller $3\boldsymbol \times 3$ kernels trained using STDP-based synaptic weight updates performed over a mini-batch of input patterns. Our deep SpiCNN, consisting of two convolutional layers trained using the unsupervised convolutional STDP learning methodology, achieved classification accuracies of 91.1% and 97.6%, respectively, for inferring handwritten digits from the MNIST data set and a subset of natural images from the Caltech data set.

107 citations


Journal ArticleDOI
TL;DR: Zhang et al. as mentioned in this paper proposed a brain-inspired cognitive model with attention for self-driving cars, which consists of a convolutional neural network for simulating the human visual cortex, a cognitive map to describe the relationships between objects in a complex traffic scene, and a recurrent neural network, which is combined with the real-time updated cognitive map, to implement the attention mechanism and long short term memory.
Abstract: The perception-driven approach and end-to-end system are two major vision-based frameworks for self-driving cars. However, it is difficult to introduce attention and historical information into the autonomous driving process, which are essential for achieving human-like driving in these two methods. In this paper, we propose a novel model for self-driving cars called the brain-inspired cognitive model with attention. This model comprises three parts: 1) a convolutional neural network for simulating the human visual cortex; 2) a cognitive map to describe the relationships between objects in a complex traffic scene; and 3) a recurrent neural network, which is combined with the real-time updated cognitive map to implement the attention mechanism and long-short term memory. An advantage of our model is that it can accurately solve three tasks simultaneously: 1) detecting the free space and boundaries for the current and adjacent lanes; 2) estimating the distances to obstacles and vehicle attitude; and 3) learning the driving behavior and decision-making process of a human driver. Importantly, the proposed model can accept external navigation instructions during an end-to-end driving process. To evaluate the model, we built a large-scale road-vehicle dataset containing over 40 000 labeled road images captured by three cameras placed on our self-driving car. Moreover, human driving activities and vehicle states were recorded at the same time.

80 citations


Journal ArticleDOI
TL;DR: In order to solve the difficulty of single AUV full coverage task of large water range, the multi-AUVs full coverage discrete and centralized programming is proposed based on GBNN algorithm which is a new developed tool with small amount of calculation and high efficiency.
Abstract: For the complete coverage path planning of autonomous underwater vehicles (AUVs), a new strategy with Glasius bio-inspired neural network (GBNN) algorithm with discrete and centralized programming is proposed. The basic modeling for multi-AUVs complete coverage problem based on grid map and neural network is discussed first. Then, the design for single AUV complete coverage is introduced based on GBNN algorithm which is a new developed tool with small amount of calculation and high efficiency. In order to solve the difficulty of single AUV full coverage task of large water range, the multi-AUV full coverage discrete and centralized programming is proposed based on GBNN algorithm. The simulation experiment is conducted to confirm that through the proposed algorithm, multi-AUVs can plan reasonable and collision-free coverage path and reach full coverage on the same task area with division of labor and cooperation.

76 citations


Journal ArticleDOI
TL;DR: A biologically inspired framework for robot learning based on demonstrations is proposed, which combines the dynamic movement primitive with the Gaussian mixture model (GMM) to integrate the features of multiple demonstrations, and a neural network-based controller is developed for the robot to track the generated motions.
Abstract: In this paper, we propose a biologically inspired framework for robot learning based on demonstrations. The dynamic movement primitive (DMP), which is motivated by neurobiology and human behavior, is employed to model a robotic motion that is generalizable. However, the DMP method can only be used to handle a single demonstration. To enable the robot to learn from multiple demonstrations, the DMP is combined with the Gaussian mixture model (GMM) to integrate the features of multiple demonstrations, where the conventional GMM is further replaced by the fuzzy GMM (FGMM) to improve the fitting performance. Also, a novel regression algorithm for FGMM is derived to retrieve the nonlinear term of the DMP. Additionally, a neural network-based controller is developed for the robot to track the generated motions. In this network, the cerebellar model articulation controller is employed to compensate for the unknown robot dynamics. The experiments have been performed on a Baxter robot to demonstrate the effectiveness of the proposed methods.

73 citations


Journal ArticleDOI
TL;DR: This paper proposes a brain–computer interface (BCI)-based teleoperation strategy for a dual-arm robot carrying a common object by multifingered hands based on motor imagery of the human brain, which utilizes common spatial pattern method to analyze the filtered electroencephalograph signals.
Abstract: This paper proposes a brain–computer interface (BCI)-based teleoperation strategy for a dual-arm robot carrying a common object by multifingered hands. The BCI is based on motor imagery of the human brain, which utilizes common spatial pattern method to analyze the filtered electroencephalograph signals. Human intentions can be recognized and classified into the corresponding reference commands in task space for the robot according to phenomena of event-related synchronization/desynchronization, such that the object manipulation tasks guided by human user’s mind can be achieved. Subsequently, a concise dynamics consisting of the dynamics of the robotic arms and the geometrical constraints between the end-effectors and the object is formulated for the coordinated dual arm. To achieve optimization motion in the task space, a redundancy resolution at velocity level has been implemented through neural-dynamics optimization. Extensive experiments have been made by a number of subjects, and the results were provided to demonstrate the effectiveness of the proposed control strategy.

70 citations


Journal ArticleDOI
TL;DR: A comprehensive survey of current hand motion sensing technologies and analysis approaches in recent emerging applications and introduces the state-of-the-art analysis methods, with a particular focus on the multimodal handmotion sensing and analysis.
Abstract: Human hand motion (HHM) analysis is an essential research topic in recent applications, especially for dexterous robot hand manipulation learning from human hand skills. It provides important information about the gestures, tactile, speed, and contact force, captured via multiple sensing technologies. This paper introduces a comprehensive survey of current hand motion sensing technologies and analysis approaches in recent emerging applications. First, the nature of HHMs is discussed in terms of simple motions, such as grasps and gestures, and complex motions, e.g., in-hand manipulations and regrasps; second, different techniques for hand motion sensing, including contact-based and noncontact-based approaches, are discussed with comparisons with their pros and cons; then, the state-of-the-art analysis methods are introduced, with a particular focus on the multimodal hand motion sensing and analysis; finally, cutting-edge applications of hand motion analysis are reviewed, with further discussion on facing challenges and new future directions.

55 citations


Journal ArticleDOI
TL;DR: The notion of a symbol in semiotics from the humanities is introduced, to leave the very narrow idea of symbols in symbolic AI and the challenges facing the creation of cognitive systems that can be part of symbol emergence systems.
Abstract: Humans use signs, e.g., sentences in a spoken language, for communication and thought. Hence, symbol systems like language are crucial for our communication with other agents and adaptation to our real-world environment. The symbol systems we use in our human society adaptively and dynamically change over time. In the context of artificial intelligence (AI) and cognitive systems, the symbol grounding problem has been regarded as one of the central problems related to symbols . However, the symbol grounding problem was originally posed to connect symbolic AI and sensorimotor information and did not consider many interdisciplinary phenomena in human communication and dynamic symbol systems in our society, which semiotics considered. In this paper, we focus on the symbol emergence problem, addressing not only cognitive dynamics but also the dynamics of symbol systems in society, rather than the symbol grounding problem. We first introduce the notion of a symbol in semiotics from the humanities, to leave the very narrow idea of symbols in symbolic AI. Furthermore, over the years, it became more and more clear that symbol emergence has to be regarded as a multifaceted problem. Therefore, second, we review the history of the symbol emergence problem in different fields, including both biological and artificial systems, showing their mutual relations. We summarize the discussion and provide an integrative viewpoint and comprehensive overview of symbol emergence in cognitive systems. Additionally, we describe the challenges facing the creation of cognitive systems that can be part of symbol emergence systems.

54 citations


Journal ArticleDOI
TL;DR: From the comparison study of SOM algorithm, Hungarian algorithm, $k$ -means algorithm, and the proposed dual competition strategy, it can be found that the task assignment with the proposed strategy is more rational and fair.
Abstract: Task assignment is an important research topic in multiple autonomous underwater vehicle (AUV) cooperative working system. However, many studies concentrate on minimizing total distance of AUVs serving targets at different locations, and mostly do not pay attention to workload balance among inhomogeneous AUVs. What is more, most of them do not think of the effect of ocean current while distributing tasks. To solve these problems, a novel dual competition strategy based on self-organizing map (SOM) neural network is put forward. An AUV makes use of surplus sailing distance to a target when it competes with others for engaging the target. In order to fulfill a balanced task assignment among AUVs, a task balance coefficient is also proposed. Meanwhile, a hybrid path planning approach is applied to guide AUVs to reach their targets safely. The good performance of the proposed algorithm for distributing tasks among AUVs is demonstrated through simulation studies. From the comparison study of SOM algorithm, Hungarian algorithm, $k$ -means algorithm, and the proposed dual competition strategy, it can be found that the task assignment with the proposed strategy is more rational and fair.

47 citations


Journal ArticleDOI
TL;DR: Electroencephalogram (EEG) emotion recognition based on a hybrid feature extraction method in empirical mode decomposition domain combining with optimal feature selection based on sequence backward selection is proposed, which can reflect subtle information of multiscale components of unstable and nonlinear EEG signals and remove the reductant features to improve the performance of emotion recognition.
Abstract: Electroencephalogram (EEG) emotion recognition based on a hybrid feature extraction method in empirical mode decomposition domain combining with optimal feature selection based on sequence backward selection is proposed, which can reflect subtle information of multiscale components of unstable and nonlinear EEG signals and remove the reductant features to improve the performance of emotion recognition. The proposal is tested on DEAP dataset, in which the emotional states in the Valance dimension and Arousal dimension are classified by both ${K}$ -nearest neighbor and support vector machine, respectively. In the experiments, temporal windows of different length and three kinds of rhythms of EEG signal are taken into account for comparison, from which the results show that EEG signal with 1s temporal window achieves highest recognition accuracy of 86.46% in Valence dimension and 84.90% in Arousal dimension, respectively, which is superior to some state-of-the-art works. The proposed method would be applied to real-time emotion recognition in multimodal emotional communication-based humans–robots interaction system.

44 citations


Journal ArticleDOI
TL;DR: A multimodal robotic system for a specific dressing scenario—putting on a shoe, where users’ personalized inputs contribute to a much improved task success rate and smaller number of user commands, and reduced workload is described.
Abstract: Robot-assisted dressing is performed in close physical interaction with users who may have a wide range of physical characteristics and abilities. Design of user adaptive and personalized robots in this context is still indicating limited, or no consideration, of specific user-related issues. This paper describes the development of a multimodal robotic system for a specific dressing scenario—putting on a shoe, where users’ personalized inputs contribute to a much improved task success rate. We have developed: 1) user tracking, gesture recognition, and posture recognition algorithms relying on images provided by a depth camera; 2) a shoe recognition algorithm from RGB and depth images; and 3) speech recognition and text-to-speech algorithms implemented to allow verbal interaction between the robot and user. The interaction is further enhanced by calibrated recognition of the users’ pointing gestures and adjusted robot’s shoe delivery position. A series of shoe fitting experiments have been performed on two groups of users, with and without previous robot personalization, to assess how it affects the interaction performance. Our results show that the shoe fitting task with the personalized robot is completed in shorter time, with a smaller number of user commands, and reduced workload.

Journal ArticleDOI
TL;DR: A deep ${Q}$ -network (DQN) that employs a multitask learning method to localize class-specific objects and can achieve higher average precision with fewer search steps than similar methods is proposed.
Abstract: In object localization, methods based on a top-down search strategy that focus on learning a policy have been widely researched. The performance of these methods relies heavily on the policy in question. This paper proposes a deep ${Q}$ -network (DQN) that employs a multitask learning method to localize class-specific objects. This DQN agent consists of two parts, an action executor part and a terminal part. The action executor determines the action that the agent should perform and the terminal decides whether the agent has detected the target object. By taking advantage of the capability of feature learning in a multitask method, our method combines these two parts by sharing hidden layers and trains the agent using multitask learning. A detection dataset from the PASCAL visual object classes challenge 2007 was used to evaluate the proposed method, and the results show that it can achieve higher average precision with fewer search steps than similar methods.

Journal ArticleDOI
TL;DR: A wireless, noninvasive and multifunctional assistive system which integrates steady state visually evoked potential-based BCI and a robotic arm to assist patients to feed themselves and feedback from the participants demonstrates that this assistives system is able to significantly improve the quality of daily life.
Abstract: Several kinds of brain–computer interface (BCI) systems have been proposed to compensate for the lack of medical technology for assisting patients who lose the ability to use motor functions to communicate with the outside world. However, most of the proposed systems are limited by their nonportability, impracticality, and inconvenience because of the adoption of wired or invasive electroencephalography acquisition devices. Another common limitation is the shortage of functions provided because of the difficulty of integrating multiple functions into one BCI system. In this paper, we propose a wireless, noninvasive and multifunctional assistive system which integrates steady state visually evoked potential-based BCI and a robotic arm to assist patients to feed themselves. Patients are able to control the robotic arm via the BCI to serve themselves food. Three other functions: 1) video entertainment; 2) video calling; and 3) active interaction are also integrated. This is achieved by designing a functional menu and integrating multiple subsystems. A refinement decision-making mechanism is incorporated to ensure the accuracy and applicability of the system. Fifteen participants were recruited to validate the usability and performance of the system. The averaged accuracy and information transfer rate achieved is 90.91% and 24.94 bit per min, respectively. The feedback from the participants demonstrates that this assistive system is able to significantly improve the quality of daily life.

Journal ArticleDOI
TL;DR: A behavior control system for social robots in therapies with a focus on personalization and platform-independence that provides the robot an ability to behave as a personable character, which behaviors are adapted to user profiles and responses during the human–robot interaction.
Abstract: Social robots have been proven beneficial in different types of healthcare interventions. An ongoing trend is to develop (semi-)autonomous socially assistive robotic systems in healthcare context to improve the level of autonomy and reduce human workload. This paper presents a behavior control system for social robots in therapies with a focus on personalization and platform-independence. This system architecture provides the robot an ability to behave as a personable character, which behaviors are adapted to user profiles and responses during the human–robot interaction. Robot behaviors are designed at abstract levels and can be transferred to different social robot platforms. We adopt the component-based software engineering approach to implement our proposed architecture to allow for the replaceability and reusability of the developed components. We introduce three different experimental scenarios to validate the usability of our system. Results show that the system is potentially applicable to different therapies and social robots. With the component-based approach, the system can serve as a basic framework for researchers to customize and expand the system for their targeted healthcare applications.

Journal ArticleDOI
TL;DR: Experimental results show that fewer exploration is needed to obtain a high expected reward, due to the prior knowledge obtained from knowledge transfer in this novel learning framework.
Abstract: Learning skills autonomously is a particularly important ability for an autonomous robot. A promising approach is reinforcement learning (RL) where agents learn policy through interaction with its environment. One problem of RL algorithm is how to tradeoff the exploration and exploitation. Moreover, multiple tasks also make a great challenge to robot learning. In this paper, to enhance the performance of RL, a novel learning framework integrating RL with knowledge transfer is proposed. Three basic components are included: 1) probability policy reuse; 2) dynamic model learning; and 3) model-based ${Q}$ -learning. In this framework, the prelearned skills are leveraged for policy reuse and dynamic learning. In model-based ${Q}$ -learning, the Gaussian process regression is used to approximate the ${Q}$ -value function so as to suit for robot control. The prior knowledge retrieved from knowledge transfer is integrated into the model-based ${Q}$ -learning to reduce the needed learning time. Finally, a human-robot handover experiment is performed to evaluate the learning performance of this learning framework. Experiment results show that fewer exploration is needed to obtain a high expected reward, due to the prior knowledge obtained from knowledge transfer.

Journal ArticleDOI
TL;DR: This work proposes a novel canonical correlation analysis (CCA)-based multiview convolutional neural network (CNNs) framework for RGB-D object representation, using the obtained CCA projection matrices to replace the weights of feature concatenation layer at regular intervals.
Abstract: Object recognition methods based on multimodal data, color plus depth (RGB-D), usually treat each modality separately in feature extraction, which neglects implicit relations between two views and preserves noise from any view to the final representation. To address these limitations, we propose a novel canonical correlation analysis (CCA)-based multiview convolutional neural network (CNNs) framework for RGB-D object representation. The RGB and depth streams process corresponding images, respectively, then are connected by CCA module leading to a common-correlated feature space. In addition, to embed CCA into deep CNNs in a supervised manner, two different schemes are explored. One considers CCA as a regularization (CCAR) term adding to the loss function. However, solving CCA optimization directly is neither computationally efficient nor compatible with the mini-batch-based stochastic optimization. Thus, we further propose an approximation method of CCAR, using the obtained CCA projection matrices to replace the weights of feature concatenation layer at regular intervals. Such a scheme enjoys benefits of full CCAR and is efficient by amortizing its cost over many training iterations. Experiments on benchmark RGB-D object recognition datasets have shown that the proposed methods outperform most existing methods using the very same of their network architectures.

Journal ArticleDOI
TL;DR: Comparisons with state-of-the-art approaches indicate the superiority of the two-stream framework in anomaly detection, and a weighted Euclidean loss is proposed to enable the network to concentrate on moving foregrounds, thus restraining background influence.
Abstract: We propose an anomaly detection approach by learning a generative model of moving pedestrians to guarantee public safety. To resolve the existing challenges of anomaly detection in complicated definitions, complex backgrounds, and local occurrence, a weighted convolutional autoencoder-long short-term memory network is proposed to reconstruct raw data and their corresponding optical flow and then perform anomaly detection based on reconstruction errors. Unlike equally treating raw data and optical flow, a novel two-stream framework is proposed to take the reconstructed optical flow as supplementary cues that encode pedestrian motions. A weighted Euclidean loss is proposed to enable the network to concentrate on moving foregrounds, thus restraining background influence. Global-local analysis is used to jointly detect and localize local anomaly in reconstructed raw data. Final detection of anomalous events is achieved by jointly considering the results of the global-local analysis and reconstructed optical flow. Qualitative evaluations verify the effectiveness of our two-stream framework, the weighted Euclidean loss, and the global-local analysis. Moreover, comparisons with state-of-the-art approaches indicate the superiority of our approach in anomaly detection.

Journal ArticleDOI
TL;DR: A comprehensive survey is summarized by recent achievements in neuro-robotics, and some potential directions for the development of future neuro- robotics are discussed.
Abstract: Neuro-robotics systems (NRSs) is the current state-of-the-art research with the strategic alliance of neuroscience and robotics. It endows the next generation of robots with embodied intelligence to identify themselves and interact with humans and environments naturally. Therefore, it needs to study the interaction of recent breakthroughs in brain neuroscience, robotics, and artificial intelligence where smarter robots could be developed by employing neural mechanisms and understanding brain functions. Recently, more sophisticated neural mechanisms of perception, cognition, learning, and control have been decoded, which investigate how to define and develop the “brain” for future robots. In this paper, a comprehensive survey is summarized by recent achievements in neuro-robotics, and some potential directions for the development of future neuro-robotics are discussed.

Journal ArticleDOI
TL;DR: This work presents the last enhanced version of a bio-inspired reinforcement-learning (RL) modular architecture able to perform skill-to-skill knowledge transfer and called transfer expert RL (TERL) model, based on a RL actor–critic model where both actor and critic have a hierarchical structure.
Abstract: When humans learn several skills to solve multiple tasks, they exhibit an extraordinary capacity to transfer knowledge between them. We present here the last enhanced version of a bio-inspired reinforcement-learning (RL) modular architecture able to perform skill-to-skill knowledge transfer and called transfer expert RL (TERL) model. TERL architecture is based on a RL actor–critic model where both actor and critic have a hierarchical structure, inspired by the mixture-of-experts model, formed by a gating network that selects experts specializing in learning the policies or value functions of different tasks. A key feature of TERL is the capacity of its gating networks to accumulate, in parallel, evidence on the capacity of experts to solve the new tasks so as to increase the responsibility for action of the best ones. A second key feature is the use of two different responsibility signals for the experts’ functioning and learning: this allows the training of multiple experts for each task so that some of them can be later recruited to solve new tasks and avoid catastrophic interference. The utility of TERL mechanisms is shown with tests involving two simulated dynamic robot arms engaged in solving reaching tasks, in particular a planar 2-DoF arm, and a 3-D 4-DoF arm.

Journal ArticleDOI
TL;DR: An online brain–machine interface (BMI) system based on multichannel steady-state visual evoked potentials was developed, and it was able to perform band-pass filtering and visual stimuli classification using support vector machines and the kinematics redundancy scheme was developed for joint velocity optimization subject to physical constraints.
Abstract: This paper describes the brain-actuated control of a dual-arm robot performing bimanual relative motion manipulation tasks, through the adoption of relative Jacobian matrix, wherein the dual-arm robot can be considered as a single manipulator and the movements of the end effectors can be calculated by the relative motion. An online brain–machine interface (BMI) system based on multichannel steady-state visual evoked potentials was developed, and it was able to perform band-pass filtering and visual stimuli classification using support vector machines. Considering the relative motion in a constrained plane, the asymmetric bimanual manipulation can be transformed into 2-D control tasks through polar coordinate transformation such that the end effectors can achieve smooth direction-and-distance motion in the arbitrary position of the operational space. Moreover, the kinematics redundancy scheme, using online neuro-dynamics optimization, was developed for joint velocity optimization subject to physical constraints. Five individuals participated in the experiments and successfully fulfilled the given manipulation task.

Journal ArticleDOI
TL;DR: The result shows that robots could build new learning rules in a less explicit manner inspired by living creatures, and gives an alternative way to efficiently develop complex behavior control of the NeuroSnake robot.
Abstract: Neurorobotic mimics the structural and functional principles of living creature systems. Modeling a single system by robotic hardware and software has existed for decades. However, an integrated toolset studying the interaction of all systems has not been demonstrated yet. We present a hybrid neuromorphic computing paradigm to bridge this gap by combining the neurorobotics platform (NRP) with the neuromorphic snake-like robot (NeuroSnake). This paradigm encompasses the virtual models, neuromorphic sensing and computing capabilities, and physical bio-inspired bodies, with which an experimenter can design and execute both in-silico and in-vivo robotic experimentation easily. The NRP is a public Web-based platform for easily testing brain models with virtual bodies and environments. The NeuroSnake is a bio-inspired robot equipped with a silico-retina sensor and neuromorphic computer for power-efficiency applications. We illustrate the efficiencies of our paradigm with an easy designing of a visual pursuit experiment in the NRP. We study two automatic behavior learning tasks which are further integrated into a complex task of semi-autonomous pole climbing. The result shows that robots could build new learning rules in a less explicit manner inspired by living creatures. Our method gives an alternative way to efficiently develop complex behavior control of the ro As spiking neural network is a bio-inspired neural network and the NeuroSnake robot is equipped with a spike-based silicon retina camera, the control system can be easily implemented via spiking neurons simulated on neuromorphic hardware, such as SpiNNaker.bot.

Journal ArticleDOI
TL;DR: A brain–computer interface (BCI), based on steady-state visually evoked potentials, exploits multivariate synchronization index classification algorithm to analyze the human electroencephalograph (EEG) signals so that the human intention can be recognized accurately, and then the EEG-based motion commands are produced for the mobile robot.
Abstract: In this paper, a brain–computer interface (BCI)-based navigation and control strategy is developed for a mobile robot in indoor environments. It combines the simultaneous localization and mapping to achieve the navigation and positioning for a mobile robot in indoor environments, where the RGB landmarks are regarded as the environmental features learned by the FastSLAM algorithm. The online BCI, based on steady-state visually evoked potentials, exploits multivariate synchronization index classification algorithm to analyze the human electroencephalograph (EEG) signals so that the human intention can be recognized accurately, and then the EEG-based motion commands are produced for the mobile robot. Probability potential field approach based on the probability density function of 2-D normal distribution is connected with the brain signals to generate a collision-free trajectory for the mobile robot. The entire system is semiautonomous, since the robot’s low level behaviors are autonomous and the stochastic navigation is executed by the BCI, and it is verified by the extensive experiments involving five volunteers. All the participants can successfully tele-operate the mobile robot, and the experimental results have verified the effectiveness of the proposed approach.

Journal ArticleDOI
TL;DR: A spectrum of approaches are outlined which mitigate these disadvantages to enable an O-D thermal IR camera equipped with a mobile robot to track a human in a variety of environments and conditions to enable intelligent perception in unmanned systems.
Abstract: We investigate human behavior-based target tracking from omni-directional (O-D) thermal images for intelligent perception in unmanned systems. Current target tracking approaches are primarily focused on perspective visual and infrared (IR) band, as well as O-D visual band tracking. The target tracking from O-D images and the use of O-D thermal vision have not been adequately addressed. Thermal O-D images provide a number of advantages over other passive sensor modalities such as illumination invariance, wide field-of-view, ease of identifying heat-emitting objects, and long term tracking without interruption. Unfortunately, thermal O-D sensors have not yet been widely used due to the following disadvantages: low resolution, low frame rates, high cost, sensor noise, and an increase in tracking time. This paper outlines a spectrum of approaches which mitigate these disadvantages to enable an O-D thermal IR camera equipped with a mobile robot to track a human in a variety of environments and conditions. The curve matched Kalman filter is used for tracking a human target based on the behavioral movement of the human and maximum a posteriori (MAP)-based estimation is extended for the human tracking as long term which provides a faster prediction. The benefits to using our MAP-based method are decreasing the prediction time of a target’s position and increasing the accuracy of prediction of the next target position based on the target’s previous behavior while increasing the tracking view and lighting conditions via the view from O-D IR camera.

Journal ArticleDOI
Siyu Yu1, Nanning Zheng1, Yongqiang Ma1, Hao Wu1, Badong Chen1 
TL;DR: Wang et al. as mentioned in this paper proposed a correlation network (CorrNet) framework that could be flexibly combined with diverse pattern representation models to decode cognitive activity patterns, which can achieve significant improvement in brain decoding over comparable methods.
Abstract: Brain decoding is a hot spot in cognitive science, which focuses on reconstructing perceptual images from brain activities. Analyzing the correlations of collected data from human brain activities and representing activity patterns are two key problems in brain decoding based on functional magnetic resonance imaging signals. However, existing correlation analysis methods mainly focus on the strength information of voxel, which reveals functional connectivity in the cerebral cortex. They tend to neglect the structural information that implies the intracortical or intrinsic connections; that is, structural connectivity. Hence, the effective connectivity inferred by these methods is relatively unilateral. Therefore, we propose in this paper a correlation network (CorrNet) framework that could be flexibly combined with diverse pattern representation models. In the CorrNet framework, the topological correlation is introduced to reveal structural information. Rich correlations can be obtained, which contribute to specifying the underlying effective connectivity. We also combine the CorrNet framework with a linear support vector machine and a dynamic evolving spike neuron network for pattern representation separately, thus provide a novel method for decoding cognitive activity patterns. Experimental results verify the reliability and robustness of our CorrNet framework, and demonstrate that the new method can achieve significant improvement in brain decoding over comparable methods.

Journal ArticleDOI
Fuchun Sun1, Wenchang Zhang1, Jianhua Chen1, Hang Wu, Chuanqi Tan1, Weihua Su 
TL;DR: A new shared control method based on fused fuzzy Petri nets (PNs) for combining the robot automatic control (AC) and the brain-actuated control (BCI) and improves safety and robustness by comparing with AC.
Abstract: It is hard to grasp objects based on brain–computer interface (BCI) by brain-actuated robot arm and hand due to its high degree of freedom. Shared control strategy and hybrid BCI are research trends to solve this control problem of brain-actuated discrete event system. We propose a new shared control method based on fused fuzzy Petri nets (PNs) for combining the robot automatic control (AC) and the brain-actuated control. This method takes the advantages of both fuzzy control and PNs such as easy modeling, robustness, and effectiveness. Both MATLAB simulation test and Barrett robot hand practical experiments show that the proposed method performs much better than AC or BCI control independently. In the online BCI practical experiment, the user successfully control the Barrett robot hand to grasp object avoiding obstacle in whole ten random scenes by our shared control method and hybrid BCI. Compared with BCI control, the user needs not to synchronously work according to the specific paradigm in the whole process. Meanwhile, our method improves safety and robustness by comparing with AC.

Journal ArticleDOI
TL;DR: This paper investigates the active visual-tactile cross-modal matching problem which is formulated as retrieving the relevant sample in unlabeled gallery visual dataset in response to the tactile query sample, and designs a shared dictionary learning model which can simultaneously learn the projection subspace and the latent shared dictionary for the visual and tactile measurements.
Abstract: Tactile and visual modalities frequently occur in cognitive robotics. Their matching problem is of highly interesting in many practical scenarios since it provides different properties about objects. In this paper, we investigate the active visual-tactile cross-modal matching problem which is formulated as retrieving the relevant sample in unlabeled gallery visual dataset in response to the tactile query sample. Such a problem exhibits a nontrivial challenge that there does not exist sample-to-sample pairing relation between tactile and visual modalities. To this end, we design a shared dictionary learning model which can simultaneously learn the projection subspace and the latent shared dictionary for the visual and tactile measurements. In addition, an optimization algorithm is developed to effectively solve the shared dictionary learning problem. Based on the obtained solution, the visual-tactile cross-modal matching algorithm can be easily developed. Finally, we perform experimental validations on the PHAC-2 datasets to show the effectiveness of the proposed visual-tactile cross-modal matching framework and method.

Journal ArticleDOI
Jiru Wang1, Vui Ann Shim1, Rui Yan1, Huajin Tang1, Fuchun Sun2 
TL;DR: This paper proposes an automatic object searching framework for a mobile robot equipped with a single RGB-D camera that requires the robot to perform obstacle avoidance and automatically search and approach the target object.
Abstract: Automatic object searching is one of the essential skills for domestic robots to operate in unstructured human environments. It involves concatenation of several capabilities, including object identification, obstacle avoidance, path planning, and navigation. In this paper, we propose an automatic object searching framework for a mobile robot equipped with a single RGB-D camera. The obstacle avoidance is achieved by a behavior learning algorithm based on deep belief networks. The target object is recognized using scale-invariant feature transform descriptors and the relative position between the target and mobile robot is estimated from the RGB-D data. Subsequently, the mobile robot makes a path planning to the target location using an improved bug-based algorithm. The framework is tested in indoor environments and requires the robot to perform obstacle avoidance and automatically search and approach the target object. The results indicate that the system is collision free and reliable in performing searching tasks. This system’s functions make itself have the potential of being used for local navigation in unstructured environments.

Journal ArticleDOI
TL;DR: The main contribution of this paper is to show how the EP controller realizes dynamic balance with no input about inertial parameter identification for the robot or terrain information estimation and is simplified for stability problems and is easy to use in practice.
Abstract: The goal of this paper is to present a force control scheme for quadrupedal locomotion by adopting observations of biological motor behavior. Specifically, based on the equilibrium point (EP) hypothesis, we set up a bio-inspired EP controller in Cartesian space. The proposed EP controller modifies the EP trajectories appropriately over time from two perspectives, which can ensure stable interactions and system equilibrium. One perspective is directly compensating for the posture angle error based on torso posture compensation. The other is based on the foot force tracking algorithm and admittance model. The main contribution of this paper is to show how the EP controller realizes dynamic balance with no input about inertial parameter identification for the robot or terrain information estimation. Overall, the EP control scheme is simplified for stability problems and is easy to use in practice. Finally, we carried out a series of simulations and experiments to evaluate the effectiveness of the EP control algorithm. The results demonstrate that the proposed controller may improve dynamic stability and realize compliance performance.

Journal ArticleDOI
TL;DR: A novel SSL method which incorporates the first-order and the second-order moments of the features in an intermediate layer of the discriminator, called mean and variance feature matching GAN (MVFM-GAN), which achieves superior performance in semi-supervised classification tasks and a better stability of GAN training.
Abstract: The improved generative adversarial network (improved GAN) is a successful method using a generative adversarial model to solve the problem of semi-supervised learning (SSL). The improved GAN learns a generator with the technique of mean feature matching which penalizes the discrepancy of the first-order moment of the latent features. To better describe common attributes of a distribution, this paper proposes a novel SSL method which incorporates the first-order and the second-order moments of the features in an intermediate layer of the discriminator, called mean and variance feature matching GAN (MVFM-GAN). To capture more precisely the data manifold, not only the mean but also the variance is used in the latent feature learning. Compared with improved GAN and other traditional methods, MVFM-GAN achieves superior performance in semi-supervised classification tasks and a better stability of GAN training, particularly in the cases when the number of labeled samples is low. It shows a comparable performance with the state-of-the-art methods on several benchmark data sets. As a byproduct of the novel approach, MVFM-GAN generates realistic images of good visual quality.

Journal ArticleDOI
TL;DR: This paper further develops previous studies by enabling regrasp planning using stable object poses on complex structures by enabling the development of a planner that has higher performance onregrasping than the previous ones.
Abstract: Using regrasp planning, a robot could pick up an object, place it down to an intermediate stable state, and reorient the object into certain poses by grasping and picking it up again Regrasp is an important skill when a robot cannot reorient the object directly with one grasp due to kinematic constraints and collisions It uses intermediate object states to release, regrasp, and reorient objects In our previous work, we developed regrasp algorithms considering intermediate stable states on simple fixtures like a flat table surface This paper further develops our previous studies by enabling regrasp planning using stable object poses on complex structures The complex structures have high variety of contact elements They not only provide flat surface supports but also point supports, line supports, and a combination of them In detail, the developed regrasp planner includes two parallel processes One is a dynamic simulator that computes immediate stable poses on given supporting structures The other builds a regrasp graph using the stable states and finds a sequence of reorient motion by searching the graph We performed thousands of simulation and real-world verification to analyze the performance of the developed planner We conclude that our planner has higher performance on regrasping than the previous ones The number of candidate regrasp sequences increased and the lengths became shorter We also compared simulation results with real-world executions, and gave suggestions on the selection of supporting structures considering real-world implementations