Multigraph Transformer for Free-Hand Sketch Recognition
Reads0
Chats0
TLDR
Peng et al. as mentioned in this paper proposed a graph neural network (GNN) for learning representations of sketches from multiple graphs, which simultaneously capture global and local geometric stroke structures as well as temporal information.Abstract:Â
Learning meaningful representations of free-hand sketches remains a challenging task given the signal sparsity and the high-level abstraction of sketches. Existing techniques have focused on exploiting either the static nature of sketches with convolutional neural networks (CNNs) or the temporal sequential property with recurrent neural networks (RNNs). In this work, we propose a new representation of sketches as multiple sparsely connected graphs. We design a novel graph neural network (GNN), the multigraph transformer (MGT), for learning representations of sketches from multiple graphs, which simultaneously capture global and local geometric stroke structures as well as temporal information. We report extensive numerical experiments on a sketch recognition task to demonstrate the performance of the proposed approach. Particularly, MGT applied on 414k sketches from Google QuickDraw: 1) achieves a small recognition gap to the CNN-based performance upper bound (72.80% versus 74.22%) and infers faster than the CNN competitors and 2) outperforms all RNN-based models by a significant margin. To the best of our knowledge, this is the first work proposing to represent sketches as graphs and apply GNNs for sketch recognition. Code and trained models are available at https://github.com/PengBoXiangShang/multigraph_transformer.read more
Citations
More filters
Proceedings ArticleDOI
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting
Ayan Kumar Bhunia,Pinaki Nath Chowdhury,Yongxin Yang,Timothy M. Hospedales,Tao Xiang,Yi-Zhe Song +5 more
TL;DR: In this article, a cross-modal translation pre-text task for self-supervised feature learning is proposed, where vectorization and rasterization are used to map image space to vector coordinates and vector coordinates to image space, respectively.
Journal ArticleDOI
Deep Learning for Free-Hand Sketch: A Survey
TL;DR: A comprehensive survey of the deep learning techniques oriented at free-hand sketch data, and the applications that they enable can be found in this paper , where the authors highlight the essential differences between sketch data and other data modalities, e.g., natural photos.
Posted Content
SketchLattice: Latticed Representation for Sketch Manipulation
TL;DR: Sketch lattice as discussed by the authors is a set of points sampled from the pixelative format of the sketch using a lattice graph, which preserves the structural cues that vector data provides, and can be used for image-to-sketch synthesis.
Journal ArticleDOI
Multistage Spatio-Temporal Networks for Robust Sketch Recognition
TL;DR: Wang et al. as discussed by the authors proposed a novel framework that aims at the multi-stage interactions and refinements of spatial and temporal features, where given a sketch represented by a stroke array, they first generate a temporal-enriched image (TEI), which is a pseudo-color image retaining the temporal order of strokes, to overcome the difficulty of CNNs in leveraging temporal information.
Proceedings ArticleDOI
Leveraging Unlabeled Data for Sketch-based Understanding
TL;DR: In this article , the authors present a study about the use of unlabeled data to improve a sketch-based model and evaluate variations of VAE and semi-supervised VAE, and present an extension of BYOL to deal with sketches.