scispace - formally typeset

Posted ContentDOI

A cooperative mobile robot and manipulator system (Co-MRMS) for transport and lay-up of fibre plies in modern composite material manufacture

TL;DR: In the proposed system, marker-based and Fourier transform-based machine vision approaches are used to achieve high accuracy capability in localisation and fibre orientation detection respectively and a particle based approach is adopted to model material deformation during manipulation within robotic simulations.

AbstractComposite materials are widely used in industry due to their light weight and specific performance. Currently, composite manufacturing mainly relies on manual labour and individual skills, especially in transport and lay-up processes, which are time consuming and prone to errors. As part of a preliminary investigation into the feasibility of deploying autonomous robotics for composite manufacturing, this paper presents a case study that investigates a cooperative mobile robot and manipulator system (Co-MRMS) for material transport and composite lay-up, which mainly comprises a mobile robot, a fixed-base manipulator and a machine vision sub-system. In the proposed system, marker-based and Fourier transform-based machine vision approaches are used to achieve high accuracy capability in localisation and fibre orientation detection respectively. Moreover, a particle-based approach is adopted to model material deformation during manipulation within robotic simulations. As a case study, a vacuum suction-based end-effector model is developed to deal with sagging effects and to quickly evaluate different gripper designs, comprising of an array of multiple suction cups. Comprehensive simulations and physical experiments, conducted with a 6-DOF serial manipulator and a two-wheeled differential drive mobile robot, demonstrate the efficient interaction and high performance of the Co-MRMS for autonomous material transportation, material localisation, fibre orientation detection and grasping of deformable material. Additionally, the experimental results verify that the presented machine vision approach achieves high accuracy in localisation (the root mean square error is 4.04 mm) and fibre orientation detection (the root mean square error is 1.84∘) and enables dealing with uncertainties such as the shape and size of fibre plies.

Topics: Mobile robot (59%), Machine vision (54%), Serial manipulator (52%), Robotics (50%)

Summary (3 min read)

1 Introduction

  • Due to the interesting properties and high strength-to-weight ratio, the applications of composite materials have raised considerably in the last decades [1, 2].
  • The method presented in [22] used a fibre reflection model to measure fibre orientation from an image and achieved good accuracies and robustness for different types of surfaces.
  • Compared to previous works, this research addresses specific challenges that arise from the introduction of different robots that must be coordinated along with the complex set of tasks covering transport, detection, grasping and placement of deformable material for composite manufacturing applications.
  • Another issue of automated handling composite material is end-effector design.
  • Until now, a number of grippers, such as grid gripper and suction cup gripper, have been designed.

2 The proposed system and approach

  • 1 Framework of the cooperative mobile robot and manipulator system (Co-MRMS) From a hardware perspective, the proposed Co-MRMS involves four components: a mobile robot, a fixed-base robotic manipulator, a vision system and a host PC.
  • Aided by the vision system, the estimated position and orientation of the raw material are sent to the fixed-base robot manipulator via the host PC.
  • The method presented above for modelling deformable objects in CoppeliaSim enables the stiffness of the overall material to be adjusted by tweaking two different types of model parameters: principle moments of inertia and individual primitive cuboids dimensions.
  • The modified suction cup gripper with four suction cups, provides a useful simulation component for quickly evaluating different gripper designs comprising of an arrangement of multiple suction cups.
  • This provides the Co-MRMS with a higher accuracy estimation of the position of the fibre material, which does not accumulate error over time.

2.3.1 Localisation approach

  • As shown in Fig. 4, this work uses a single ArUco vision marker for material localisation material.
  • The Suzuki algorithm [40] is then used to extract the contours, which are reconstructed by the Douglas-Peucker algorithm [41].
  • Cells belonging to the border of the image carry a value of 0, while all inner cells are analysed to obtain the internal encoding, which corresponds to a 6x6 internal grid area.
  • To improve the accuracy of the marker detection, the corners of the marker are refined through subpixel interpolation.
  • Using this approach, the position of the material can be determined robustly regardless of the size and shape of the material.

2.3.2 Fibre orientation detection approach

  • This is due to the material being anisotropic, meaning it provides varying strength along different directions across the material.
  • In order to make sure that the plies are layered as designed, strict requirements are imposed for the orientation of each layer of fibres to obtain the expected Title Suppressed Due to Excessive Length 11 composite parts.
  • The Fourier Transform is applied for fibre orientation analysis, where an image is converted into the frequency domain to obtain its spatial frequency components.
  • Then the Fourier transform is applied to obtain the frequency domain image.
  • Here curve fitting is achieved through the use of the least squares line fitting method.

3 Experimental Setup

  • The composite material used here is a small sheet of fabric prepreg.
  • Localisation of the material and fibre orientation detection are achieved through the use of a spotlight mounted together with the camera to produce strong reflections from the fibres of the material.
  • In addition, the deformable object was modelled in CoppeliaSim by leveraging its support for the simulation of dynamic behaviours, which is achieved through the Bullet 2.78 physics engine.
  • For the physical implementation, the ITRA toolbox [46], developed for the control of KUKA robots, provided the interface for directly sending actuation commands from Matlab to the KUKA robot controller unit for manipulator control, while ROS provided the interface for the actuation of the Turtlebot3 Burger.
  • The vision system relied upon images captured by a webcam mounted on the end-effector of the manipulator to observe the environment.

4 Performance Evaluation

  • To validate the developed system, several experiments were conducted to test the capabilities of the Co-MRMS.
  • Initially, simulation-based experiments were carried out according to the proposed approaches in Section 2 and the accuracy of the vision system was assessed.
  • The Co-MRMS, which employs a KUKA KR90 R3100 industrial fixed-base manipulator and a Turtlebot3 Burger differential drive mobile robot, was firstly modelled in CoppeliaSim to verify the performance in fulfilling the transportation and lay-up task of the proposed system.
  • With the material position data obtained from machine vision system and wheel odometry, the localisation accuracy could be evaluated through Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE).
  • In the second experiment, the fibre orientation detection algorithm was evaluated by comparing the output of the algorithm against the ground truth.

4.2.1 System interaction behaviour evaluation

  • The cooperative system interaction behaviour was evaluated by physical experiments, of which a set of execution routines consisting of five active phases and two idle phases were obtained.
  • The duration of this phase varies according to the start point, goal point and the subsequent path to move between these two points.
  • After a brief pause where all systems remain idle to indicate that the mobile robot has reached its destination, the host PC sends the wheel odometry estimation of the mobile robot’s position as a target command to drive the manipulator towards the approximate location of the material (phase 2).
  • This facilitates the placement of the material in a controlled orientation during grasping operations by ensuring that the fibre direction is always aligned with the z axis rotation of the end-effector.
  • This experiment demonstrated the capability of the integrated system to correct any manipulator positional offset error that arises from wheel slippage of the mobile robot through higher accuracy estimation provided by machine vision.

4.2.2 Machine vision system accuracy evaluation

  • To measure the accuracy of the machine vision algorithms in the real world, two additional experiments were conducted.
  • The first experiment was used to quantify the errors in the measured position of the mobile robot using the vision-based localisation algorithm and wheel odometry.
  • This experiment was conducted 20 times for statistical significance.
  • A sample piece of composite material was placed in a fixed position in the workspace of the manipulator while the endeffector was positioned directly above the centre of the material with their rotation axes aligned at 0◦.
  • The investigation shows that the closer the true fibre orientation is to 0◦, the higher the accuracy in fibre orientation detection.

5 Discussions

  • This section discusses the obtained experimental results.
  • Firstly, it should be noted that the simulated trials incorporating the manipulation actions for grasping the material has so far not been included in the physical trials due to the lack of vacuum suction hardware.
  • Instead, the educational mobile robot platform Turtlebot 3 Burger was adopted for the investigations conducted.
  • Thus, additional development work is necessary to implement the proposed system framework onto an industrial standard set of hardware to validate the proposed system.
  • Title Suppressed Due to Excessive Length 21.

6 Conclusions

  • A cooperative mobile robot and manipulator system (Co-MRMS), which comprised of a fixed-base manipulator, an autonomous mobile robot and a machine vision sub-system, was developed as a promising strategy for autonomous material transfer and handling tasks to advance composite manufacturing.
  • To demonstrate the feasibility and effectiveness of the proposed Co-MRMS, comprehensive simulations and physical experiments have been conducted.
  • In conclusion, by exploiting the availability of wheel odometry and integrating this with machine vision algorithms within the proposed Co-MRMS, it is possible to implement a flexible system that provides autonomous material transportation and sufficiently-accurate material handling capabilities that extend beyond what is currently adopted in the industry.
  • This research was funded by the Route to Impact Program 2019–2020 [grant no.: AFRC CATP 1469 R2I-Academy] and supported by the Advanced Forming Research Centre (University of Strathclyde), Lightweight Manufacturing Centre (University of Strathclyde) and Control Robotics Intelligence Group (Nanyang Technological University, Singapore).
  • The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Did you find this useful? Give us your feedback

...read more

Content maybe subject to copyright    Report

A Cooperative Mobile Robot and Manipulator
System (Co-MRMS) for Transport and Lay-up of
Fibre Plies in Modern Composite Material
Manufacture
Manman Yang ( manman.yang@strath.ac.uk )
University of Strathclyde
Leijian Yu
University of Strathclyde
Cuebong Wong
National Nuclear Laboratory Ltd
Carmelo Mineo
University of Palermo: Universidad de Palermo
Erfu Yang
University of Strathclyde https://orcid.org/0000-0003-1813-5950
Iain Bomphray
Lightweight manufacturing centre
Ruoyu Huang
Lightweight manufacturing centre
Research Article
Keywords: cooperative robots , composite material manufacturing , machine vision , transport and lay-up
Posted Date: June 8th, 2021
DOI: https://doi.org/10.21203/rs.3.rs-566792/v1
License: This work is licensed under a Creative Commons Attribution 4.0 International License. 
Read Full License
Version of Record: A version of this preprint was published at The International Journal of Advanced
Manufacturing Technology on November 23rd, 2021. See the published version at
https://doi.org/10.1007/s00170-021-08342-2.

Journal of Intelligent & Robotic Systems manuscript No.
(will be inserted by the editor)
A Cooperative Mobile Robot and Manipulator System
(Co-MRMS) for Transport and Lay-up of fibre Plies in
Modern Composite Material Manufacture
Manman Yang
1
· Leijian Yu
1
· Cuebong
Wong
2
· Carmelo Mineo
3
· Erfu Yang
1
· Iain
Bomphray
4
· Ruoyu Huang
4
Received: date / Accepted: date
Abstract Composite materials are widely used in industry due to their light weight
and specific performance. Currently, composite manufacturing mainly relies on man-
ual labour and individual skills, especially in transport and lay-up processes, which
are time-consuming and prone to errors. As part of a preliminary investigation into
the feasibility of deploying autonomous robotics for composite manufacturing, this
paper investigates a cooperative mobile robot and manipulator system (Co-MRMS)
for material transport and composite lay-up, which mainly comprises a mobile robot,
a fixed-base manipulator and a machine vision sub-system. In the proposed system,
marker-based and Fourier transform-based machine vision approaches are used to
achieve high accuracy capability in localisation and fibre orientation detection respec-
tively. Moreover, a particle based approach is adopted to model material deformation
during manipulation within robotic simulations. As a case study, a vacuum suction
based end-effector model is developed to deal with sagging effects and to quickly
evaluate different gripper designs, comprising of an array of multiple suction cups.
Comprehensive simulations and physical experiments, conducted with a 6-DOF se-
rial manipulator and a two-wheeled differential drive mobile robot, demonstrate the
efficient interaction and high performance of the Co-MRMS for autonomous mate-
rial transportation, material localisation, fibre orientation detection and grasping of
deformable material. Additionally, the experimental results verify that the presented
machine vision approach achieves high accuracy in localisation (the root mean square
error is 4.04 mm) and fibre orientation detection (the root mean square error is 1.84
).
1
Design, Manufacturing & Engineering Management,University of Strathclyde, G1 1XJ Glasgow, United
Kingdom.
2
National Nuclear Laboratory Ltd. Havelock Rd, CA14 3YQ Workington, United Kingdom.
3
Department of Engineering, University of Palermo, Viale delle Scienze, Edificio 8, 90128 Palermo, Italy.
4
Lightweight Manufacturing Centre, National Manufacturing Institute Scotland, University of Strathclyde,
G1 1XJ Glasgow, United Kingdom.
Corresponding author:
Erfu Yang
E-mail: erfu.yang@strath.ac.uk

2 Manman Yang
1
et al.
Keywords cooperative robots · composite material manufacturing · machine vision ·
transport and lay-up
1 Introduction
Due to the interesting properties and high strength-to-weight ratio, the applications
of composite materials have raised considerably in the last decades [1, 2]. They are
usually made of multiple plies of fibres (e.g. carbon, glass and/or synthetic fibres),
layered up in alternating orientations and held together by resin [3]. Therefore, the
laying-up of fibre plies is the fundamental manufacturing phase in the production
of composite materials. It is usually performed by human operators, who handle
and transport the raw materials, making composite manufacturing time-consuming,
labour intensive and prone-to-errors. The demand for the phasing in of robotic so-
lutions to improve process efficiency and increase operator safety has grown signifi-
cantly. Automated Tape Laying (ATL) [4] and Automated fibre Placement (AFP) [5]
are two popular automated technologies employed in automotive lay-up of compos-
ite material. However, limited by the heavy cost of specialised equipment and low
flexibility, they are only suitable for making small composite parts [6]. Up to now,
investigations on the use of commercially available robotic platforms for composite
lay-up are on the rise in composite manufacturing.
Previous works have investigated the viability of using robotic systems in ad-
vanced composite manufacturing by exploiting the flexibility of robots to meet the
stringent demands of manufacturing processes. In [7] and [8], complete systems for
handling and laying up prepreg on a mould were developed. Robotic workcells were
demonstrated with different modules. Bjornsson et al. [9] surveyed pick-and-place
systems in automated composite handling with regards to handling strategy, grip-
ping technology and reconfigurability etc. This survey indicated that it is hard to find
generic design principle and the best solution for handling raw materials for compos-
ite manufacture depends on the specific case study. Schuster et al. [10, 11] demon-
strated how cooperative robotic manipulators can execute the automated draping pro-
cess of large composite plies in physical experiments. Similar research has been done
by Deden et al., who also addressed the complete handling process from path plan-
ning and end-effector design to ply detection [12]. Szcesny et al.[13] proposed an
innovative approach for automated composite ply placement by employing three in-
dustrial manipulators, where two of them were equipped with grippers for material
grasping and the third manipulates a mounted compaction roller for layer compres-
sion. A comparable hybrid robot cell was developed by Malhan et al. [14, 15], where
rapid refinement of online grasping trajectories was studied. Despite these advances,
cooperative/hybrid robotic systems involving mobile robot platforms and fixed-base
robotic manipulators have received little attention in the context of advanced com-
posite manufacturing.
Due to the requirement of accurate localisation and fibre orientation detection, an
efficient vision system is of great importance for autonomous robotic system in ad-
vanced composite manufacturing. Fibre orientation detection is challenging due to the
high surface reflectivity and fine weaving of the material, and thus it has still predom-

Title Suppressed Due to Excessive Length 3
inantly been accomplished manually in practice [16, 17]. Traditional machine vision
methods for fibre orientation detection of textiles prefer to utilize diffused lighting
[18], such as diffuse dome [19] and flat diffuse [20] illumination measuring tech-
niques. Polarisation model approaches have been particularly popular for measuring
fibre orientation, where contrast between textile features such as fibres and seams are
used to identify the structure of the material relative to the camera [21]. The method
presented in [22] used a fibre reflection model to measure fibre orientation from an
image and achieved good accuracies and robustness for different types of surfaces.
However, when considering the specific application of advanced composite manufac-
turing, changes in lighting conditions are often unavoidable because of the moving
shadow of the robot arm cast on the material. The integration of vision systems with
robotics was considered by only few of the previous works. This means systems are
inflexible as they are unable to cope with dynamic variations within advanced com-
posite manufacturing processes.
In composite manufacturing, material transport and composite lay-up have not
been integrated into a single autonomous robotic system, which is challenging due
to the many technologies involved, including path planning, material detection and
localisation, etc. Achieving this requires the development of a strategy that combines
different modules in a flexible system and provides autonomous material transporta-
tion and sufficiently-accurate material handling capabilities. This paper presents a
case study on robotic material transportation and composite lay-up, which is based on
a real-world scenario commonly found in advanced composite manufacturing. Com-
pared to previous works, this research addresses specific challenges that arise from
the introduction of different robots that must be coordinated along with the complex
set of tasks covering transport, detection, grasping and placement of deformable ma-
terial for composite manufacturing applications. The aim of this research is to conduct
a pilot study on the feasibility of deploying a cooperative robotic system to perform
a series of tasks in composite material manufacturing. Therefore, a cooperative mo-
bile robot and manipulator system (Co-MRMS), which consists of an autonomous
mobile robot, a fixed-base manipulator and a machine vision sub-system is presented
in this paper. The mobile robot transports the material autonomously to a predefined
position within the working range of the fixed-base manipulator. The machine vision
sub-system then detects the location of the material and estimates the fibre orienta-
tion to enable the manipulator to accurately handle the material. This is achieved by
employing an ArUco marker detection algorithm [23] to compute the position of the
material, and a Fourier transform-based algorithm [24] combined with a least squares
line fitting method [25] to calculate the material’s fibre orientation. Afterwards, the
manipulator accurately grasps the material and places it onto a mould. Simulated tri-
als and physical experiments are conducted to verify the cooperation behaviours of
the Co-MRMS and quantify the accuracy of the vision system.
In addition, modelling of flexible deformable objects has been one of the most
researched topics in robotics, such as cables and fabrics. To realistically simulate
the interactive behaviour between robot actions (i.e. grasping and transfer actions)
and material deformation, various techniques do exist to model deformable objects.
In [26], recent advancement of different types of flexible deformable object mod-
elling for robotic manipulation, such as physical-based and mass-spring modelling,

4 Manman Yang
1
et al.
was reviewed. Moreover, the approaches of building up deformable object mod-
els were presented. Researchers in [27] established a model for deformable cables
and investigated robotic cable assembling, addressing collision detection issues. This
study adopted particle-based modelling approach [28] to model material deformation
within simulation when composite material is grasped and transferred by a manipu-
lator. Another issue of automated handling composite material is end-effector design.
Until now, a number of grippers, such as grid gripper and suction cup gripper, have
been designed. Suction cup grippers could handle deformable objects without dam-
aging the material and are flexible enough to drape different shapes of composite
material to flat or curved moulds. Gerngross et al. [29] developed suction cup based
grippers for handling prepregs in Offline programming. The solution of automated
handling dry textiles to double curvature mould were verified both in offline pro-
gramming environment and an industrial scale manufacturing demonstrator. Ellek-
ilde et al. [30] designed a novel draping tool with up to 120 suction cups, which has
been tested on draping large aircraft part prepreg. Krogh et al. [31] researched the
moving trajectories of suction cup gripper for draping plies with establishing cable
model. Therefore, a vacuum suction-based end-effector model is developed in this
work to simulate sagging effects during grasping, which provides a useful simulation
tool for quickly evaluating different gripper designs comprising of an arrangement of
multiple suction cups.
The remaining parts of the paper are organised as follows. First, the framework
of the Co-MRMS, the modelling strategy for the interaction with deformable ob-
jects and machine vision approaches are described in Section 2. Then, the details of
the experimental setup are outlined in Section 3. Section 4 discusses the Co-MRMS
evaluation through physical experiments, while Section 5 is devoted to a discussion
on the findings, limitations and future directions of the work. Finally, the conclusions
are provided in Section 6.
2 The proposed system and approach
2.1 Framework of the cooperative mobile robot and manipulator system
(Co-MRMS)
From a hardware perspective, the proposed Co-MRMS involves four components: a
mobile robot, a fixed-base robotic manipulator, a vision system and a host PC. The
framework of the Co-MRMS is presented in Fig. 1. The mobile robot is responsi-
ble for transporting the composite material from a given starting location within the
work shop floor (e.g. the storage area) to the robotic manipulator. Aided by the vi-
sion system, the estimated position and orientation of the raw material are sent to the
fixed-base robot manipulator via the host PC. The manipulator is used for grasping
each fibre ply and placing it correctly according to the designed lay-up manufactur-
ing specifications. Robotic path planning for both robots was implemented in MAT-
LAB® [32]. Image processing algorithms were developed by using OpenCV [33],
an open source computer vision and machine vision software library that provides
a common infrastructure for computer vision applications and accelerates the devel-

References
More filters



Journal ArticleDOI
TL;DR: There is a natural uncertainty principle between detection and localization performance, which are the two main goals, and with this principle a single operator shape is derived which is optimal at any scale.
Abstract: This paper describes a computational approach to edge detection. The success of the approach depends on the definition of a comprehensive set of goals for the computation of edge points. These goals must be precise enough to delimit the desired behavior of the detector while making minimal assumptions about the form of the solution. We define detection and localization criteria for a class of edges, and present mathematical forms for these criteria as functionals on the operator impulse response. A third criterion is then added to ensure that the detector has only one response to a single edge. We use the criteria in numerical optimization to derive detectors for several common image features, including step edges. On specializing the analysis to step edges, we find that there is a natural uncertainty principle between detection and localization performance, which are the two main goals. With this principle we derive a single operator shape which is optimal at any scale. The optimal detector has a simple approximate implementation in which edges are marked at maxima in gradient magnitude of a Gaussian-smoothed image. We extend this simple detector using operators of several widths to cope with different signal-to-noise ratios in the image. We present a general method, called feature synthesis, for the fine-to-coarse integration of information from operators at different scales. Finally we show that step edge detector performance improves considerably as the operator point spread function is extended along the edge.

26,639 citations


Proceedings Article
01 Jan 2009
TL;DR: This paper discusses how ROS relates to existing robot software frameworks, and briefly overview some of the available application software which uses ROS.
Abstract: This paper gives an overview of ROS, an opensource robot operating system. ROS is not an operating system in the traditional sense of process management and scheduling; rather, it provides a structured communications layer above the host operating systems of a heterogenous compute cluster. In this paper, we discuss how ROS relates to existing robot software frameworks, and briefly overview some of the available application software which uses ROS.

7,367 citations


Book
01 Jan 1965
Abstract: 1 Introduction 2 Groundwork 3 Convolution 4 Notation for Some Useful Functions 5 The Impulse Symbol 6 The Basic Theorems 7 Obtaining Transforms 8 The Two Domains 9 Waveforms, Spectra, Filters and Linearity 10 Sampling and Series 11 The Discrete Fourier Transform and the FFT 12 The Discrete Hartley Transform 13 Relatives of the Fourier Transform 14 The Laplace Transform 15 Antennas and Optics 16 Applications in Statistics 17 Random Waveforms and Noise 18 Heat Conduction and Diffusion 19 Dynamic Power Spectra 20 Tables of sinc x, sinc2x, and exp(-71x2) 21 Solutions to Selected Problems 22 Pictorial Dictionary of Fourier Transforms 23 The Life of Joseph Fourier

5,709 citations


Frequently Asked Questions (2)
Q1. What are the contributions mentioned in the paper "A cooperative mobile robot and manipulator system (co-mrms) for transport and lay-up of fibre plies in modern composite material manufacture" ?

As part of a preliminary investigation into the feasibility of deploying autonomous robotics for composite manufacturing, this paper investigates a cooperative mobile robot and manipulator system ( Co-MRMS ) for material transport and composite lay-up, which mainly comprises a mobile robot, a fixed-base manipulator and a machine vision sub-system. As a case study, a vacuum suction based end-effector model is developed to deal with sagging effects and to quickly evaluate different gripper designs, comprising of an array of multiple suction cups. 

Future work will focus on validating the proposed system on industrial standard platforms and improving the system e. g. integrating a vacuum gripper, quantifying system efficiency, extending the work to multiple plies and developing a method for draping correction. In conclusion, by exploiting the availability of wheel odometry and integrating this with machine vision algorithms within the proposed Co-MRMS, it is possible to implement a flexible system that provides autonomous material transportation and sufficiently-accurate material handling capabilities that extend beyond what is currently adopted in the industry.