In this paper, we propose a real-time online shopper behavior analysis system consisting of two modules which simultaneously predicts the visitor’s shopping intent and Web site abandonment likelihood. In the first module, we predict the purchasing intention of the visitor using aggregated pageview data kept track during the visit along with some session and user information. The extracted features are fed to random forest (RF), support vector machines (SVMs), and multilayer perceptron (MLP) classifiers as input. We use oversampling and feature selection preprocessing steps to improve the performance and scalability of the classifiers. The results show that MLP that is calculated using resilient backpropagation algorithm with weight backtracking produces significantly higher accuracy and F1 Score than RF and SVM. Another finding is that although clickstream data obtained from the navigation path followed during the online visit convey important information about the purchasing intention of the visitor, combining them with session information-based features that possess unique information about the purchasing interest improves the success rate of the system. In the second module, using only sequential clickstream data, we train a long short-term memory-based recurrent neural network that generates a sigmoid output showing the probability estimate of visitor’s intention to leave the site without finalizing the transaction in a prediction horizon. The modules are used together to determine the visitors which have purchasing intention but are likely to leave the site in the prediction horizon and take actions accordingly to improve the Web site abandonment and purchase conversion rates. Our findings support the feasibility of accurate and scalable purchasing intention prediction for virtual shopping environment using clickstream and session information data.

Real-time prediction of online shoppers’ purchasing intention using multilayer perceptron and LSTM recurrent neural networks

Data mining applications in accounting: A review of the literature and organizing framework

Sequential Tests of Statistical Hypotheses.

Web scraping is a technique to extract information from various web documents automatically. It retrieves the related contents based on the query, aggregates and transforms the data from an unstructured format into a structured representation. Text classification becomes a vital phase to summarize the data and in categorizing the webpages adequately. In this article, using effective web scraping methodologies, the data is initially extracted from websites, then transformed into a structured form. Based on the keywords from the data, the documents are classified and labeled. A recursive feature elimination technique is applied to the data to select the best candidate feature subset. The final data-set trained with standard machine learning algorithms. The proposed model performs well on classifying the documents from the extracted data with a better accuracy rate.

Personalized Content Extraction and Text Classification Using Effective Web Scraping Techniques

E-commerce offers various merchandise for selling and purchasing with frequent transactions and commodity flows. An accurate prediction of customer needs and optimized allocation of goods is required for cost reduction. The existing solutions have significant errors and are unsuitable for addressing warehouse needs and allocation. That is why businesses cannot respond to customer demands promptly, as they need accurate and reliable demand forecasting. Therefore, this paper proposes spatial feature fusion and grouping strategies based on multimodal data and builds a neural network prediction model for e-commodity demand. The designed model extracts order sequence features, consumer emotional features, and facial value features from multimodal data from e-commerce products. Then, a bidirectional long short-term memory network- (BiLSTM-) based grouping strategy is proposed. The proposed strategy fully learns the contextual semantics of time series data while reducing the influence of other features on the group’s local features. The output features of multimodal data are highly spatially correlated, and this paper employs the spatial dimension fusion strategy for feature fusion. This strategy effectively obtains the deep spatial relations among multimodal data by integrating the features of each column in each group across spatial dimensions. Finally, the proposed model’s prediction effect is tested using e-commerce dataset. The experimental results demonstrate the proposed algorithm’s effectiveness and superiority.

/pdf/multimodal-data-guided-spatial-feature-fusion-and-grouping-2a34k8o0u2.pdf

Multimodal Data Guided Spatial Feature Fusion and Grouping Strategy for E-Commerce Commodity Demand Forecasting

The paper addresses the problem of e-customer behavior characterization based on Web server log data. We describe user sessions with the number of session features and aim to identify the features indicating a high probability of making a purchase for two customer groups: traditional customers and innovative customers. We discuss our approach aimed at assessing a purchase probability in a user session depending on categories of viewed products and session features. We apply association rule mining to real online bookstore data. The results show differences in factors indicating a high purchase probability in session for both customer types. The discovered association rules allow us to formulate some predictions for the online store, e.g. that a logged user who has viewed only traditional, printed books, has been staying in the store from 10 to 25 min, and has opened between 30 and 75 pages, will decide to confirm a purchase with the probability of more than 92 %.

/pdf/using-association-rules-to-assess-purchase-probability-in-1xljkqydk4.pdf

Using association rules to assess purchase probability in online stores

Bot recognition in a Web store: An approach based on unsupervised learning

A large part of Web traffic on e-commerce sites is generated not by human users but by Internet robots: search engine crawlers, shopping bots, hacking bots, etc. In practice, not all robots, especially the malicious ones, disclose their identities to a Web server and thus there is a need to develop methods for their detection and identification. This paper proposes the application of a Bayesian approach to robot detection based on characteristics of user sessions. The method is applied to the Web traffic from a real e-commerce site. Results show that the classification model based on the cluster analysis with the Ward's method and the weighted Euclidean metric is very effective in robot detection, even obtaining accuracy of above 90%.

Detection of Internet robots using a Bayesian approach

Recent studies reported that about half of Web users nowadays are intelligent agents (Web bots). Many bots are impersonators operating at a very high sophistication level, trying to emulate navigational behaviors of legitimate users (humans). Moreover, bot technology continues to evolve which makes bot detection even harder. To deal with this problem, many advanced methods for differentiating bots from humans have been proposed, a large part of which relies on supervised machine learning techniques. In this paper, we propose a novel approach to identify various profiles of bots and humans which combines feature selection and unsupervised learning of HTTP-level traffic patterns to develop a user session classification model. Session clustering is performed with the agglomerative Information Bottleneck (aIB) algorithm, as well as with some other reference algorithms. The model is then used to classify new sessions to one of the profiles and to label the sessions as performed by bots or humans. An extensive experimental study, based on real server log data, demonstrates the ability of aIB clustering to distinguish user profiles and confirms high performance of the classification model in terms of accuracy, F1, recall, and precision.

Identifying legitimate Web users and bots with different traffic profiles — an Information Bottleneck approach

A significant volume of Web traffic nowadays can be attributed to robots. Although some of them, e.g., search-engine crawlers, perform useful tasks on a website, others may be malicious and should be banned. Consequently, there is a growing need to identify bots and to characterize their behavior. This paper investigates the share of bot-generated traffic on an e-commerce site and studies differences in bots' and humans' session-based traffic by analyzing data recorded in Web server log files. Results show that both kinds of sessions reveal different characteristics, including the session duration, the number of pages visited in session, the number of requests, the volume of data transferred, the mean time per page, the number of images per page, and the percentage of pages with unassigned referrers.

/pdf/analysis-of-aggregated-bot-and-human-traffic-on-e-commerce-26quczg6qf.pdf

Grażyna Suchacka

Papers

Using association rules to assess purchase probability in online stores

Bot recognition in a Web store: An approach based on unsupervised learning

Detection of Internet robots using a Bayesian approach

Identifying legitimate Web users and bots with different traffic profiles — an Information Bottleneck approach

Analysis of aggregated bot and human traffic on e-commerce site