Learning meaningful representations of free-hand sketches remains a challenging task given the signal sparsity and the high-level abstraction of sketches. Existing techniques have focused on exploiting either the static nature of sketches with Convolutional Neural Networks (CNNs) or the temporal sequential property with Recurrent Neural Networks (RNNs). In this work, we propose a new representation of sketches as multiple sparsely connected graphs. We design a novel Graph Neural Network (GNN), the Multi-Graph Transformer (MGT), for learning representations of sketches from multiple graphs which simultaneously capture global and local geometric stroke structures, as well as temporal information. We report extensive numerical experiments on a sketch recognition task to demonstrate the performance of the proposed approach. Particularly, MGT applied on 414k sketches from Google QuickDraw: (i) achieves small recognition gap to the CNN-based performance upper bound (72.80% vs. 74.22%), and (ii) outperforms all RNN-based models by a significant margin. To the best of our knowledge, this is the first work proposing to represent sketches as graphs and apply GNNs for sketch recognition. Code and trained models are available at this https URL.

Multi-Graph Transformer for Free-Hand Sketch Recognition

Free-hand sketches are highly illustrative, and have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly popular. The progress of deep learning has immensely benefited free-hand sketch research and applications. This paper presents a comprehensive survey of the deep learning techniques oriented at free-hand sketch data, and the applications that they enable. The main contents of this survey include: (i) A discussion of the intrinsic traits and unique challenges of free-hand sketch, to highlight the essential differences between sketch data and other data modalities, e.g., natural photos. (ii) A review of the developments of free-hand sketch research in the deep learning era, by surveying existing datasets, research topics, and the state-of-the-art methods through a detailed taxonomy and experimental evaluation. (iii) Promotion of future work via a discussion of bottlenecks, open problems, and potential research directions for the community. Finally, to support future sketch research and applications, we contribute TorchSketch -- the first sketch-oriented open-source deep learning library, which is built on PyTorch and available at this https URL.

Deep Learning for Free-Hand Sketch: A Survey and A Toolbox

Automatic data abstraction is an important capability for both benchmarking machine intelligence and supporting summarization applications. In the former one asks whether a machine can `understand' enough about the meaning of input data to produce a meaningful but more compact abstraction. In the latter this capability is exploited for saving space or human time by summarizing the essence of input data. In this paper we study a general reinforcement learning based framework for learning to abstract sequential data in a goal-driven way. The ability to define different abstraction goals uniquely allows different aspects of the input data to be preserved according to the ultimate purpose of the abstraction. Our reinforcement learning objective does not require human-defined examples of ideal abstraction. Importantly our model processes the input sequence holistically without being constrained by the original input order. Our framework is also domain agnostic -- we demonstrate applications to sketch, video and text data and achieve promising results in all domains.

/pdf/goal-driven-sequential-data-abstraction-3du61vcy7i.pdf

Goal-Driven Sequential Data Abstraction

Free-hand sketches are highly illustrative, and have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly popular. The progress of deep learning has immensely benefited free-hand sketch research and applications. This paper presents a comprehensive survey of the deep learning techniques oriented at free-hand sketch data, and the applications that they enable. The main contents of this survey include: (i) A discussion of the intrinsic traits and unique challenges of free-hand sketch, to highlight the essential differences between sketch data and other data modalities, e.g., natural photos. (ii) A review of the developments of free-hand sketch research in the deep learning era, by surveying existing datasets, research topics, and the state-of-the-art methods through a detailed taxonomy and experimental evaluation. (iii) Promotion of future work via a discussion of bottlenecks, open problems, and potential research directions for the community. 

/pdf/deep-learning-for-free-hand-sketch-a-survey-2cd1b3pb.pdf

Deep Learning for Free-Hand Sketch: A Survey

This paper presents a novel method for matching line segments in images based on pair-wise geometric constraints and matching redundancy. In this study, pairs of line segments satisfying angle and distance constraints are used as matching primitives. To ensure that each extracted line segment is paired with another line segment, the search region of each line segment is gradually grown until it is paired. Initial pair-to-pair correspondences between two images are established using four pair-wise constraints; next, line-to-line correspondences are obtained. To effectively solve the matching conflict in the results, a method of recording the result of line pair matching based on a double-layer matrix is proposed. Based on the double-layer matrix, an effective checking method for the line matching results based on the collinearity constraint and matching redundancy is presented. It fully utilizes redundancy information and considers the collinearity of fragmented line segments. Further, it can effectively separate correct and incorrect matches from the one-to-many, many-to-one, and many-to-many matching results. The proposed method was tested on 12 image pairs from a benchmark of matched-lines, and compared with other state-of-the-art methods. The results demonstrate the superiority of the proposed method due to its higher accuracy and greater recall in challenging cases.

Robust line feature matching based on pair-wise geometric constraints and matching redundancy

Line matching based on line-points invariant and local homography

Recognizing freehand sketches with high arbitrariness is greatly challenging. Most existing methods either ignore the geometric characteristics or treat sketches as handwritten characters with fixed structural ordering. Consequently, they can hardly yield high recognition performance even though sophisticated learning techniques are employed. In this paper, we propose a sequential deep learning strategy that combines both shape and texture features. A coded shape descriptor is exploited to characterize the geometry of sketch strokes with high flexibility, while the outputs of constitutional neural networks (CNN) are taken as the abstract texture feature. We develop dual deep networks with memorable gated recurrent units (GRUs), and sequentially feed these two types of features into the dual networks, respectively. These dual networks enable the feature fusion by another gated recurrent unit (GRU), and thus accurately recognize sketches invariant to stroke ordering. The experiments on the TU-Berlin data set show that our method outperforms the average of human and state-of-the-art algorithms even when significant shape and appearance variations occur.

Sequential Dual Deep Learning with Shape and Texture Features for Sketch Recognition

Recognizing freehand sketches with high arbitrariness is such a great challenge that the automatic recognition rate has reached a ceiling in recent years. In this paper, we explicitly explore the shape properties of sketches, which has almost been neglected before in the context of deep learning, and propose a sequential dual learning strategy that combines both shape and texture features. We devise a two-stage recurrent neural network to balance these two types of features. Our architecture also considers stroke orders of sketches to reduce the intra-class variations of input features. Extensive experiments on the TU-Berlin benchmark set show that our method achieves over 90% recognition rate for the first time on this task, outperforming both humans and state-of-the-art algorithms by over 19 and 7.5 percentage points, respectively. Especially, our approach can distinguish the sketches with similar textures but different shapes more effectively than recent deep networks. Based on the proposed method, we develop an on-line sketch retrieval and imitation application to teach children or adults to draw. The application is available as Sketch.Draw.

Coupling Deep Textural and Shape Features for Sketch Recognition

Background Azithromycin (AZM) is one of the most common broad-spectrum antibiotics. However, drug resistance is increasing and combination therapy has attracted great attention. AZM is usually combined with traditional Chinese medicine (TCM) preparations with heat-clearing and detoxifying effects, including Kumu injection (KM) made from Picrasma quassioides (D. Don) Benn. Purpose The present study aimed to investigate synergistic antimicrobial and anti-inflammatory activities of KM plus AZM with the aim of understanding the mechanism of clinical efficacy of combination regimens. Methods Seven common bacterial clinical isolates and LPS-induced RAW 264.7 cells were used for assay of in vitro potency. The minimum inhibitory concentration (MIC) was determined for each drug, followed by synergy testing through the checkerboard method and fractional inhibitory concentration index (FICI) for quantifying combined antibacterial effects. The rat model of Klebsiella pneumoniae-induced pneumonia was developed and subjected to various drug treatments, namely, AZM, KM, or AZM plus KM, intravenously administered at 75 mg/kg once a day for one week. The combination effects then were evaluated according to pharmacodynamics and pharmacokinetic assessments. Results KM-AZM combination synergistically inhibits in vitro growth of all the test standard strains except Pseudomonas aeruginosa and also the drug-resistant strains of Staphylococcus aureus, Streptococcus pneumoniae, Shigella dysenteriae, Klebsiella pneumoniae, and Escherichia coli. Despite an additive effect against NO, KM plus AZM at an equal dose could synergistically suppress overrelease of the inflammatory cytokines TNF-α and IL-6 by LPS-induced RAW 264.7 cells. The combination significantly inhibited the proliferation of K. pneumoniae in the rat lungs, mainly by inactivating MAPKs and NF-κB signaling pathways. KM-AZM combination caused a onefold increase in apparent distribution volume of AZM, along with a significant decrease of AZM level in the livers and heart for pharmacokinetics. Conclusion KM-AZM combination displayed synergistic antibacterial and anti-inflammatory effects beneficial to the therapeutic potential against bacterial infection.

/pdf/combination-therapy-with-tcm-preparation-kumu-injection-and-2j3r0qh3.pdf

Meiyu Yu

Papers

Line matching based on line-points invariant and local homography

Sequential Dual Deep Learning with Shape and Texture Features for Sketch Recognition

Coupling Deep Textural and Shape Features for Sketch Recognition

Combination Therapy with TCM Preparation Kumu Injection and Azithromycin against Bacterial Infection and Inflammation: In Vitro and In Vivo