Reading information in news feed apps has become a kind of popular content consumption in recent years. However, there are contradictory conclusions about the recommendation strategies. Although some previous research has studied the algorithms to improve the accuracy of the combined of relevancy and diversity strategies, how to compromise them according to user attitude and behavior from individual level is limited. This study aims to solve the dilemma at a theoretical level, and the authors find that the degree of multi-context use is a key boundary. Specifically, 1) the location-based recommendation method may lose effectiveness when the degree of a multi-context is high. 2) The users who prefer to use news feed systems in various contexts in their daily life are more likely to read diverse information. The authors suppose that with the increase of the degree of multi-context use, the location-based recommendation may trigger users perceived privacy threat, which will reduce their satisfaction and participation intention.

/pdf/relevancy-or-diversity-recommendation-strategy-based-on-the-2zd7thhv.pdf

Relevancy or Diversity?: Recommendation Strategy Based on the Degree of Multi-Context Use of News Feed Users

Reinforcement learning is a promising paradigm for learning robot control, allowing complex control policies to be learned without requiring a dynamics model. However, even state of the art algorithms can be difficult to tune for optimum performance. We propose employing an ensemble of multiple reinforcement learning agents, each with a different set of hyperparameters, along with a mechanism for choosing the best performing set(s) on-line. In the literature, the ensemble technique is used to improve performance in general, but the current work specifically addresses decreasing the hyperparameter tuning effort. Furthermore, our approach targets on-line learning on a single robotic system, and does not require running multiple simulators in parallel. Although the idea is generic, the Deep Deterministic Policy Gradient was the model chosen, being a representative deep learning actor-critic method with good performance in continuous action settings but known high variance. We compare our online weighted q-ensemble approach to q-average ensemble strategies addressed in literature using alternate policy training, as well as online training, demonstrating the advantage of the new approach in eliminating hyperparameter tuning. The applicability to real-world systems was validated in common robotic benchmark environments: the bipedal robot half cheetah and the swimmer. Online Weighted Q-Ensemble presented overall lower variance and superior results when compared with q-average ensembles using randomized parameterizations.

Online Weighted Q-Ensembles for Reduced Hyperparameter Tuning in Reinforcement Learning

Safe Deep Reinforcement Learning (SDRL) has been studied in many domains. In previous work, based on Augmented Lagrangian Method, we proposed DDPG-ALM algorithm in an industrial oil pump case study to learn a safe controller. In this paper, we apply our algorithm to a more general platform, Safety Gym, to test the performance of our algorithm. To solve the issue of sparse cost in Safety Gym, we propose the notion of State Shaping which aggregates safe states to pay more attention to the unsafe states. Furthermore, to pay more attention to safety-related experience, we combine the advantages of Prioritized Experience Replay and Double Replay Buffer and propose a method called Safe Prioritized Experience Replay (SPER). Our experiments show that, State Shaping and SPER can deal with the sparse cost well and provide safer policies. Ablation studies show that State Shaping reduces average episode cost of all evaluated trajectories from 2.67 to 0.45 and SPER reduces from 0.45 to 0.20 while PER only reduces to 0.30. Moreover, our DDPG-ALM based algorithm not only automatically finds a balance between reward and cost, but also converges faster than other existing benchmark algorithms.

Tackling sparse cost in safe reinforcement learning for obstacle avoidance

Off-Policy Deep Reinforcement Learning (DRL) algorithms such as Deep Deterministic Policy Gradient (DDPG) has been used to teach intelligent agents to solve complicated problems in continuous space-action environments. Several methods have been successfully applied to increase the training performance and achieve better speed and stability for these algorithms. Such as experience replay to selecting a batch of transactions of the replay memory buffer. However, working with environments with sparse reward function is a challenge for these algorithms and causes them to reduce these algorithms' performance. This research intends to make the transaction selection process more efficient by increasing the likelihood of selecting important transactions from the replay memory buffer. Our proposed method works better with a sparse reward function or, in particular, with environments that have termination conditions. We are using a secondary replay memory buffer that stores more critical transactions. In the training process, transactions are select in both the first replay buffer and the secondary replay buffer. We also use a parallel environment to asynchronously execute and fill the primary replay buffer and the secondary replay buffer. This method will help us to get better performance and stability. Finally, we evaluate our proposed approach to the Crawler model, one of the Unity ML-Agent tasks with sparse reward function, against DDPG and AE-DDPG.

Duplicated Replay Buffer for Asynchronous Deep Deterministic Policy Gradient

Recently, interactive recommendation systems based on reinforcement learning have been attended by researchers due to the consider recommendation procedure as a dynamic process and update the recommendation model based on immediate user feedback, which is neglected in traditional methods. The existing works have two significant drawbacks. Firstly, inefficient ranking function to produce the Top-N recommendation list. Secondly, focusing on recommendation accuracy and inattention to other evaluation metrics such as diversity. This paper proposes a deep reinforcement learning based recommendation system by utilizing Actor-Critic architecture to model dynamic users' interaction with the recommender agent and maximize the expected longterm reward. Furthermore, we propose utilizing Spotify's ANNoy algorithm to find the most similar items to generated action by actor-network. After that, the Total Diversity Effect Ranking algorithm is used to generate the recommendation items concerning relevancy and diversity. Moreover, we apply positional encoding to compute representations of the user's interaction sequence without using sequence-aligned recurrent neural networks. Extensive experiments on the MovieLens dataset demonstrate that our proposed model is able to generate a diverse while relevance recommendation list based on the user's preferences.

Improving ranking function and diversification in interactive recommendation systems based on deep reinforcement learning

Recently, interactive recommendation systems based on reinforcement learning have been attended by researchers due to the consider recommendation procedure as a dynamic process and update the recommendation model based on immediate user feedback, which is neglected in traditional methods. The existing works have two significant drawbacks. Firstly, inefficient ranking function to produce the Top-N recommendation list. Secondly, focusing on recommendation accuracy and inattention to other evaluation metrics such as diversity. This paper proposes a deep reinforcement learning based recommendation system by utilizing Actor-Critic architecture to model dynamic users' interaction with the recommender agent and maximize the expected long-term reward. Furthermore, we propose utilizing Spotify's ANNoy algorithm to find the most similar items to generated action by actor-network. After that, the Total Diversity Effect Ranking algorithm is used to generate the recommendations concerning relevancy and diversity. Moreover, we apply positional encoding to compute representations of the user's interaction sequence without using sequence-aligned recurrent neural networks. Extensive experiments on the MovieLens dataset demonstrate that our proposed model is able to generate a diverse while relevance recommendation list based on the user's preferences.

Seyed Mohammad Seyed Motehayeri

Papers

Duplicated Replay Buffer for Asynchronous Deep Deterministic Policy Gradient

Improving ranking function and diversification in interactive recommendation systems based on deep reinforcement learning

D2RLIR : an improved and diversified ranking function in interactive recommendation systems based on deep reinforcement learning