Text-based retrieval systems have been popular, but content-based retrieval systems have gained widespread acceptance in recent years to directly retrieve diverse media based on their visual content, such as color, texture, and shape. Among many content-based retrieval systems, sketch-based media retrieval systems have attracted attention recently with the proliferation of tablet PCs and smart mobile devices. Sketch-based retrieval requires the user to draw a freehand sketch query, but freehand drawing can be challenging for those with limited drawing skills. This degrades retrieval performance, since successful retrieval depends on the quality of the sketch query image drawn by the user. To address this issue, we propose a real-time stroke guidance for freehand sketch retrieval that continuously displays next-stroke shadow sketches on the canvas based on the user's step-by-step partial strokes. We train a stroke guidance network that learns the mapping between the step-wise stroke relations to predict the user's next stroke. The proposed stroke guidance for freehand sketch retrieval system runs on a five step next-stroke prediction model that identifies candidate next-stroke sketches from a database of millions of sketches. The system retrieves variable number of sketch object classes at different drawing stages. During the initial sketching stage, diverse drawing possibilities are covered by retrieving multiple sketch classes; as the sketching progresses, the intended sketch class is narrowed down to one. Deep binary hashing is employed for efficient similarity matching of relevant next-stroke sketches. We extend the Google QuickDraw dataset to create a five step sketch stroke database. Qualitative and quantitative experiments are conducted to verify the effectiveness of the proposed system, which can be utilized for drawing guidance, tracing, and sketch retrieval. Tracing refers to the act of copying the shadowed line of a guiding image by drawing over its lines.

SketchHelper: Real-Time Stroke Guidance for Freehand Sketch Retrieval

PURPOSE Manual brain tumor segmentation is a challenging task that requires the use of machine learning techniques. One of the machine learning techniques that has been given much attention is the convolutional neural network (CNN). The performance of the CNN can be enhanced by combining other data analysis tools such as wavelet transform. MATERIALS AND METHODS In this study, one of the famous implementations of CNN, a fully convolutional network (FCN), was used in brain tumor segmentation and its architecture was enhanced by wavelet transform. In this combination, a wavelet transform was used as a complementary and enhancing tool for CNN in brain tumor segmentation. RESULTS Comparing the performance of basic FCN architecture against the wavelet-enhanced form revealed a remarkable superiority of enhanced architecture in brain tumor segmentation tasks. CONCLUSION Using mathematical functions and enhancing tools such as wavelet transform and other mathematical functions can improve the performance of CNN in any image processing task such as segmentation and classification.

/pdf/wavelet-enhanced-convolutional-neural-network-a-new-idea-in-1u1uqc83tm.pdf

Wavelet-enhanced convolutional neural network: a new idea in a deep learning paradigm.

The ability to semantically interpret hand-drawn line sketches, although very challenging, can pave way for novel applications in multimedia. We propose SKETCHPARSE, the first deep-network architecture for fully automatic parsing of freehand object sketches. SKETCHPARSE is configured as a two-level fully convolutional network. The first level contains shared layers common to all object categories. The second level contains a number of expert sub-networks. Each expert specializes in parsing sketches from object categories which contain structurally similar parts. Effectively, the two-level configuration enables our architecture to scale up efficiently as additional categories are added. We introduce a router layer which (i) relays sketch features from shared layers to the correct expert (ii) eliminates the need to manually specify object category during inference. To bypass laborious part-level annotation, we sketchify photos from semantic object-part image datasets and use them for training. Our architecture also incorporates object pose prediction as a novel auxiliary task which boosts overall performance while providing supplementary information regarding the sketch. We demonstrate SKETCHPARSE's abilities (i) on two challenging large-scale sketch datasets (ii) in parsing unseen, semantically related object categories (iii) in improving fine-grained sketch-based image retrieval. As a novel application, we also outline how SKETCHPARSE's output can be used to generate caption-style descriptions for hand-drawn sketches.

SketchParse: Towards Rich Descriptions for Poorly Drawn Sketches using Multi-Task Hierarchical Deep Networks

The ability to semantically interpret hand-drawn line sketches, although very challenging, can pave way for novel applications in multimedia. We propose SketchParse, the first deep-network architecture for fully automatic parsing of freehand object sketches. SketchParse is configured as a two-level fully convolutional network. The first level contains shared layers common to all object categories. The second level contains a number of expert sub-networks. Each expert specializes in parsing sketches from object categories which contain structurally similar parts. Effectively, the two-level configuration enables our architecture to scale up efficiently as additional categories are added. We introduce a router layer which (i) relays sketch features from shared layers to the correct expert (ii) eliminates the need to manually specify object category during inference. To bypass laborious part-level annotation, we sketchify photos from semantic object-part image datasets and use them for training. Our architecture also incorporates object pose prediction as a novel auxiliary task which boosts overall performance while providing supplementary information regarding the sketch. We demonstrate SketchParse's abilities (i) on two challenging large-scale sketch datasets (ii) in parsing unseen, semantically related object categories (iii) in improving fine-grained sketch-based image retrieval. As a novel application, we also outline how SketchParse's output can be used to generate caption-style descriptions for hand-drawn sketches.

SketchParse : Towards Rich Descriptions for Poorly Drawn Sketches using Multi-Task Hierarchical Deep Networks

Free-hand sketches are highly illustrative, and have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly popular. The progress of deep learning has immensely benefited free-hand sketch research and applications. This paper presents a comprehensive survey of the deep learning techniques oriented at free-hand sketch data, and the applications that they enable. The main contents of this survey include: (i) A discussion of the intrinsic traits and unique challenges of free-hand sketch, to highlight the essential differences between sketch data and other data modalities, e.g., natural photos. (ii) A review of the developments of free-hand sketch research in the deep learning era, by surveying existing datasets, research topics, and the state-of-the-art methods through a detailed taxonomy and experimental evaluation. (iii) Promotion of future work via a discussion of bottlenecks, open problems, and potential research directions for the community. 

/pdf/deep-learning-for-free-hand-sketch-a-survey-2cd1b3pb.pdf

Deep Learning for Free-Hand Sketch: A Survey

The paper presents a deep Convolutional Neural Network (CNN) framework for free-hand sketch recognition. One of the main challenges in free-hand sketch recognition is to increase the recognition accuracy on sketches drawn by different people. To overcome this problem, we use deep Convolutional Neural Networks (CNNs) that have dominated top results in the field of image recognition. And we use the contours of natural images for training, because sketches drawn by different people may be very different and databases of the sketch images for training are very limited. We propose a CNN training on contours that performs well on sketch recognition over different databases of the sketch images. And we make some adjustments to the contours for training and reach higher recognition accuracy. Experimental results show the effectiveness of the proposed approach.

Deep Neural Networks for Free-Hand Sketch Recognition

With the high-growing of e-commerce, more and more users have changed to buy from websites rather than in stores. To deal with mass products, the traditional text-based product search has become incompetent to meet use’s requirement. In this paper, we explore deep learning with convolutional neural networks (CNN) to resolve query’s classification, and propose an efficient approach for product image search. For a query image, we first train a CNN model of a large database containing various product images to discriminate the query’s category. Then we search similar products from the established category and utilize these visual results to parse the query with attribute. Finally we use the extracted attribute tags to finish the textual re-ranking and obtain the most relevant retrieved product list. Experimental evaluation shows that our approach significantly outperforms state of art in product image search.

Yuqi Zhang

Papers

Deep Neural Networks for Free-Hand Sketch Recognition

Product Image Search with Deep Attribute Mining and Re-ranking