Image captioning is one of the most widely discussed topic nowadays. However, most research in this area generate English caption while there are thousands of language exist around the world. With their language uniqueness, there’s a need of specific research to generate captions in those languages. Indonesia, as the largest Southeast Asian country, has its own language, which is Bahasa Indonesia. Bahasa Indonesia has been taught in various countries such as Vietnam, Australia, and Japan. In this research, we propose the attention-based image captioning model using ResNet101 as the encoder and LSTM with adaptive attention as the decoder for the Indonesian image captioning task. Adaptive attention used to decide when and at which region of the image should be attended to produce the next word. The model we used was trained with the MSCOCO and Flick30k datasets besides. Both datasets are translated manually into Bahasa by human and by using Google Translate. Our research resulted in 0.678, 0.512, 0.375, 0.274, and 0.990 for BLEU-1, BLEU-2, BLEU-3, BLEU-4, and CIDEr scores respectively. Our model also produces a similar score for the English image captioning model, which means our model capable of being equivalent to English image captioning. We also propose a new metric score by conducting a survey. The results state that 76.8% of our model’s caption results are better than validation data that has been translated using Google Translate.

Adaptive Attention Generation for Indonesian Image Captioning

Image captioning is the description of an image with natural language expressions using computer vision and natural language processing fields. Recent advances in hardware and processing power in smartphones lead the development of many image captioning applications. In this study, a novel automatic image captioning system based on the encoder-decoder approach that can be applied in smartphones is proposed. While high-level visual information is extracted with the ResNet152V2 convolutional neural network in the encoder part, the proposed decoder transforms the extracted visual information into natural expressions of the images. The proposed decoder with the multilayer gated recurrent unit structure allows generating more meaningful captions using the most relevant visual information. The proposed system has been evaluated using different performance metrics on the MSCOCO dataset and it outperforms the state-ofthe-art approaches. The proposed system is also integrated with our custom-designed Android application, named IMECA, which generates captions in offline mode unlike similar applications. Thus, image captioning is intended to be practical for more people.

Multi-GRU Based Automated Image Captioning for Smartphones

A Multilevel Transfer Learning Technique and LSTM Framework for Generating Medical Captions for Limited CT and DBT Images

Most of the data present today on Web is in the form of text. This text data is a rich source of information, but it is difficult to process because it is unstructured in nature. Various techniques have been developed in past to process this text data for information retrieval and text mining. Many machine learning techniques have been developed in past to drive valuable information from text data. However, these methods require a lot of preprocessing and feature extraction prior to processing which increases manual task. There are a lot of human interactions which not only consumes time but also makes the whole system where rigid and hard to generalize. Deep learning architectures mitigate these limitations. They require less human intervention and create better solutions than other machine learning techniques. Deep architectures are neural networks with multiple processing layers of neuron with each layer having a specific task. Various deep learning architectures have been developed till date with each having a specific task. These architectures have contributed significantly in the domain of text processing. Word2Vec and GloVe have helped to represent text in vector forms. Vector representation helps to process text easily. Architectures like CNN & autoencoders have helped to automate the task of feature extraction. On the other hand, architectures like LSTM and RNN are used for text processing. Text classification, document summarization, question answering, machine translation, caption generation, and speech recognition are some of the areas where improvements have been found due to deep learning architectures. This paper first introduces the text preprocessing techniques and feature extraction methods. Later on, a study of contributions of deep learning architectures in text processing is presented.

A Study of Deep Learning in Text Analytics

In the modern era, image captioning has become
one of the most widely required tools. Moreover, there are inbuilt
applications that generate and provide a caption for a certain
image, all these things are done with the help of deep neural
network models. The process of generating a description of an
image is called image captioning. It requires recognizing the
important objects, their attributes, and the relationships among
the objects in an image. It generates syntactically and
semantically correct sentences.In this paper, we present a deep
learning model to describe images and generate captions using
computer vision and machine translation. This paper aims to
detect different objects found in an image, recognize the
relationships between those objects and generate captions. The
dataset used is Flickr8k and the programming language used
was Python3, and an ML technique called Transfer Learning
will be implemented with the help of the Xception model, to
demonstrate the proposed experiment. This paper will also
elaborate on the functions and structure of the various Neural
networks involved. Generating image captions is an important
aspect of Computer Vision and Natural language processing.
Image caption generators can find applications in Image
segmentation as used by Facebook and Google Photos, and even
more so, its use can be extended to video frames. They will easily
automate the job of a person who has to interpret images. Not to
mention it has immense scope in helping visually impaired
people.

Image Caption Generator

The recent advances in Deep Learning based Machine Translation and Computer Vision have led to excellent Image Captioning models using advanced techniques like Deep Reinforcement Learning While these models are very accurate, these often rely on the use of expensive computation hardware making it difficult to apply these models in realtime scenarios, where their actual applications can be realised In this paper, we carefully follow some of the core concepts of Image Captioning and its common approaches and present our simplistic encoder and decoder based implementation with significant modifications and optimizations which enable us to run these models on low-end hardware of hand-held devices We also compare our results evaluated using various metrics with state-of-the-art models and analyze why and where our model trained on MSCOCO dataset lacks due to the trade-off between computation speed and quality Using the state-of-the-art TensorFlow framework by Google, we also implement a first of its kind Android application to demonstrate the realtime applicability and optimizations of our approach

Nand Kumar Bansode

Papers

Camera2Caption: A real-time image caption generator