With the rising spotlight on Data Science and Big data analytics in both private and public sectors, enormous amount of domain-specific information has been collected. Missing information is a typical issue and consequently has pulled in significant consideration. Incomplete data analysis is a practical problem, and an intriguing issue. This study presents a methodology and framework for imputing missing value and at different missing rate of variables. An effective missing data imputation model based using Long Short-Term Memory (LSTM) on numerical dataset is proposed. For the hidden layers in a deep neural network, Rectified Linear Units (ReLU) is conventionally used as an activation function. However, there are situations where progressively more complicated functions are required to tackle vanishing gradient problem. The proposed LSTM with ReLU activation function is advantageous in terms of mapping features, weight distribution and loss function. We estimate LSTM-ReLU’s hyper-parameter values and compared its average RMSE performance with another neural network LSTM model LSTM-Softmax and machine learning models like Naive Bayes and KNN. Experiments were carried out on the various datasets. Results obtained using LSTM-ReLU model outperforms LSTM-Softmax, Naive Bayes and KNN method. LSTM-ReLU achieves an average performance gain of 40% – 46% over LSTM-Softmax imputation with varied missingness value and 40% - 42% on varied hyper-parameter size. Accuracy is much greater and mean square error less than 0.02% with Naive Bayes and KNN model.

An Efficient Missing Data Imputation Model on Numerical Data

A useful technique for analyzing incomplete tables is to model the missing data mechanisms of the variables using log-linear models. In this paper, we use log-linear parametrization and propose estimation methods for arbitrary three-way and $n$-dimensional incomplete tables. All possible cases in which data on one or more of the variables may be missing are considered. We provide simple closed form estimates of expected cell counts and parameters for the various missing data models. We also obtain explicit boundary estimates under nonignorable nonresponse models. Finally, a real-life dataset is analyzed to illustrate our results for modelling and estimation in multidimensional incomplete tables.

Closed form estimates for missing counts in multidimensional incomplete tables

The analysis of incomplete contingency tables is an important problem, which is also of practical interest. In this paper, we consider boundary solutions under nonignorable nonresponse models in two-way incomplete tables with data on both variables missing. We establish a result similar to Park {\it et al.} (2014) on sufficient conditions for the occurrence of boundary solutions. We also provide a new result, which connects the forms of boundary solutions under various parameterizations of the missing data models. This result helps us to obtain the exact form of boundary solutions in the above tables, which improves a claim made in Baker {\it et al.} (1992) and avoids computational burden. A counterexample is provided to show that the sufficient conditions for the occurrence of boundary solutions are not necessary, thereby disproving a conjecture of Kim and Park (2014). Finally, we establish new necessary conditions for the occurrence of boundary solutions under nonignorable nonresponse models in square two-way incomplete tables, and show that they are not sufficient. These conditions are simple and easy to check as they depend only on the observed cell counts. They are useful and important for model selection also. Some real life data sets are analyzed to illustrate the results.

On the occurrence of boundary solutions in two-way incomplete tables

The analysis of incomplete contingency tables is a practical and an interesting problem. In this paper, we provide characterizations for the various missing mechanisms of a variable in terms of response and non-response odds for two and three dimensional incomplete tables. Log-linear parametrization and some distinctive properties of the missing data models for the above tables are discussed. All possible cases in which data on one, two or all variables may be missing are considered. We study the missingness of each variable in a model, which is more insightful for analyzing cross-classified data than the missingness of the outcome vector. For sensitivity analysis of the incomplete tables, we propose easily verifiable procedures to evaluate the missing at random (MAR), missing completely at random (MCAR) and not missing at random (NMAR) assumptions of the missing data models. These methods depend only on joint and marginal odds computed from fully and partially observed counts in the tables, respectively. Finally, some real-life datasets are analyzed to illustrate our results, which are confirmed based on simulation studies.

S. Ghosh

Papers

On the occurrence of boundary solutions in multidimensional incomplete tables

On the occurrence of boundary solutions in two-way incomplete tables