Using Multi-armed Bandit to Solve Cold-start
Problems in Recommender Systems at Telco
Hai Thanh Nguyen
1
and Anders Kofod-Petersen
1,2
1
Telenor Research
7052 Trondheim, Norway
{HaiThanh.Nguyen|Anders.Kofod-Petersen}@telenor.com
2
Department of Computer and Information Science (IDI),
Norwegian University of Science and Technology (NTNU),
7491 Trondheim, Norway
anderpe@idi.ntnu.no
Abstract. Recommending best-fit rate-plans for new users is a chal-
lenge for the Telco industry. Rate-plans differ from most traditional
products in the way that a user normally only have one product at any
given time. This, combined with no background knowledge on new users
hinders traditional recommender systems. Many Telcos today use either
trivial approaches, such as picking a random plan or the most common
plan in use. The work presented here shows that these methods perform
poorly. We propose a new approach based on the multi-armed bandit
algorithms to automatically recommend rate-plans for new users. An
experiment is conducted on two different real-world datasets from two
brands of a major international Telco operator showing promising results.
Key words: multi-armed bandit, cold-start, recommender systems, tele-
com, and rate-plan.
1 Introduction
The Telco industry do not at first glance appear to be of particular interest
from a recommender system perspective. Telcos do not commonly supply a lot
of services; most general they supply subscriptions, or rate-plans; either pre-paid
or post-paid. However, recommending the optimal rate-plans for users in general,
and new users in particular can be challenging.
Suggesting a rate-plan for a new user is a typical cold-start user problem
(following the separation suggested by Park et al., [1]). This problem has also
been identified under slightly different names, such as: the new user problem [2],
the cold start problem [3] or new-user ramp-up problem [4]. However, the fact
that a customer traditionally only has one rate-plan at any given time increases
the difficulty of this problem. Comparing this to a more traditional recommender
problem where a user-item matrix might be sparse; in this example the matrix
will be completely sparse.
To solve this cold-start problem, given the fact that no prior information on
the new user exists, one might think of a random recommendation of rate-plans.
2 Nguyen and Kofod-Petersen
However, the chance that the recommended plan be accepted by the new user is
small. In fact, given n available rate-plans the probability that a random pick-up
plan is accepted is only 1/n. We say this approach has too much randomness in
its recommendations.
Another possibility for solving this problem is to use the distribution of se-
lected plans from existing users. Concretely, it is sensible to recommend the most
popular rate-plan to the new user. By doing this, we assume that there is a fixed
distribution behind the choice of rate-plans by the new users. However, in reality
and also in the experiment below, we can observe that this is not the case. We
say this method exploits too much the most popular rate-plan.
The idea now is to have a better solution to control the randomness in the
exploration of different rate-plans while keeping the exploitation of the most
popular rate-plan at a time. This is the usual dilemma between Exploitation (of
already available knowledge) versus Exploration (of uncertainty), encountered in
sequential decision making under uncertainty problems. This has been studied
for decades in the multi-armed bandit framework. The work presented here,
attempts to tackle the cold-start user problem by recommending a plan that
will appeal to the user in question, rather than the best plan. We approach this
by applying the multi-armed bandit algorithms.
The multi-armed bandit (MAB) is a classical problem in decision theory
[5,6,7]. It models a machine with K arms, each of which has a different and
unknown distribution of rewards. The goal of the player is to repeatedly pull the
arms to maximise the expected total reward. However, since the player does not
know the distribution of rewards, he needs to explore different arms and at the
same time exploit the current optimal arm (i.e. the arm with the current highest
cumulative reward).
To evaluate our MAB approach in solving the cold-start user problem at
Telco, we conduct experiments on real-world datasets and compare it with trivial
approaches, which include the random and most popular method. Experimental
results show that our proposed approach improves upon the trivial ones.
The paper is organised as follows: Section 2 gives an overview of related
works; Section 3 provides a formal definition of the cold-start problem in the
rate-plan recommender system at Telco. We describe our proposed approaches
in Section 4. Section 5 presents some experimental results and discussions. The
paper ends with a summary of our findings and a discussion on future work.
2 Related Work
Unfortunately, there are very few examples of research regarding rate-plan rec-
ommender systems for Telco, in particular with respect to the cold-start prob-
lem. Examples include, Thomas et al., who describe how to recommend best-fit
recharges for pre-paid users [8]. Soonsiripanichkul et al., employes a na¨ıve Bayes
classifier to infer which rate-plan to suggest to existing users [9]. Both use existing
data on customers’ usage patterns and do not address the cold-start problem.
Multi-armed bandits for cold-start Problems at Telcos 3
In general, one common strategy for mitigating the cold-start user problem
is to gather demographic data. It is assumed that users who share a common
background also share a common taste in products. Examples include Lekakos
and Giaglis [10], where lifestyle information is employed. This includes age, mar-
ital status and education, as well as preferences on eight television genres. The
authors report that this approach is the most effective way of dealing with the
cold-start user problem in sparse environments.
A similar thought underlies the work by Lam et al., [11] where an aspect
model (see e.g. [12]) including age, gender and job is used. This information is
used to calculate a probability model that classifies users into user groups and
the probability how well liked an item is by this user group.
Other examples of applying demographic information for mitigating the cold-
start user problem exists, e.g. [13,14,15]. All the solutions above use similar
demographic information; most commonly age, occupation and gender. Most of
the solutions ask for less than five pieces of information. Even though five is a
comparatively small number, the user must still answer these questions. Users do
generally not like to answer a lot of questions, yet expect reasonable performance
from the first interaction with the system [16].
Zigoris and Zhang [16], suggests to use a two part Bayesian model, where
the prior probability is based on the existing user population and data likeli-
hood, which is based on the data supplied by the user. Thus, when a new user
enters the system, little is know about that user and the prior distribution is the
main contributor. As the user interacts with the system the data data likelihood
becomes more and more important. This approach performs well for cold-start
users. Other similar approaches can by found in [17], suggesting a Markov mix-
ture model, and [18] who suggests a statical user language model that integrates
an individual model, a group model and a global model.
Our study differs from previous research on the cold-start problem, as no de-
mographic information is taken into account. Only the information on selected
plans of previous users is available to the recommender engine. This assumption
makes the cold-start problems even harder to solve. However, we leave the is-
sue of collecting more information from users and how to use it for cold-start
recommender systems for future works.
3 Problem Definition
Recommending a rate-plan for a new mobile telephony user differs from tradi-
tional recommender systems. Traditionally, recommender systems are in a con-
text where users can purchase and own several products, such as books; Rate-
plans are different in the sense that one user can have any number of rate-plans,
but typically only one plan at any given time. Further, the user will typically
have the same product for an extended time period. Finally, no explicit rating
for the rate-plans exist. We call this problem the Cold Start Alternative Recom-
mendation (CSAR) problem and below is its formal definition.
4 Nguyen and Kofod-Petersen
Let U = {u
1
, . . . , u
T
} be the set of T new users. Assume that we have a set
P of n rate-plans to recommend to a new user: P = {p
1
, . . . , p
n
} where each
plan p
i
(i = 1...n) is described by m features {f
1
, . . . , f
m
}, such as price, number
of included SMS, number of voice minutes included and so on. Among k (k ≥ 1)
suggested rate-plans, the new user can only select one plan at any given time.
Assume that at a given time t a new user u
t
comes and the system recom-
mends a rate-plan p
t
without any knowledge on the new user. Depending on the
user’s needs, she will accept the offer or select another rate-plan. We want to
design an algorithm that can find a best-fit rate-plan for the new user. Let need
t
be a vector described the user’s demand: need
t
= (need
t1
, . . . , need
tm
), where
each feature need
tj
corresponds to each feature f
j
of the rate-plans. If we denote
the similarity value between the recommended plan p
t
and the actual demand
of the new user u
t
by a similarity(need
t
, p
t
), then the objective when solving
the CSAR problem is to select the rate-plans p
t
that maximizes the following so
called ”cumulative reward” (Reward) over all T new users:
Reward
T
=
T
X
t=1
(similarity(need
t
, p
t
))
The CSAR problem would be easy to solve if we knew about the user’s needs
need
t
. The task then becomes straightforward by selecting the rate-plan that
provides the maximal value of the similarity(need
t
, p
t
) over all available plans.
As mentioned, it is not possible to calculate similarity(need
t
, p
t
) since need
t
is not available. We suggest to study an approximated problem to the CSAR
problem where we consider the similarity value between the recommended plan
p
t
and the actual selection of the new user p
∗
t
. By doing this, we wish to achieve
a recommendations as close as possible to the actual choice made by the user.
The actual choice is also considered as her temporary best-fit plan. Formally, we
want to maximize the following so called ”reward”:
Reward
T
=
T
X
t=1
(similarity(p
t
, p
∗
t
))
There are many ways to define the similarity value between two vectors p
t
and
p
∗
t
. Below, we suggest to take into account the two most popular measurements
which are i ) the indicator function and ii) the correlation value.
Indicator function If we use the indicator function as the similarity mea-
surement, then the problem becomes to design an algorithm that predict the
rate-plan p
∗
t
chosen by the new user. The cummulative reward now is the fol-
lowing: Reward
(1)
T
=
P
T
t=1
(1I(p
∗
t
6= p
t
)), where 1I(p
∗
t
6= p
t
) is an indicator func-
tion which is equal to 0 if p
∗
t
6= p
t
and to 1, otherwise. To evaluate any al-
gorithm solving this problem, we can use the classical precision measurement:
P recision
T
=
1
T
Reward
(1)
T
Multi-armed bandits for cold-start Problems at Telcos 5
Correlation value In the second case, we study how similar is the recom-
mended rate-plan p
t
to the actual selection p
∗
t
of the new user in terms of the
features. Generally, when a new user purchases a rate-plan, she looks at the fea-
tures describing the different rate-plans including the recommended rate-plan p
t
.
Finally she picks up a plan p
∗
t
that we can assume is perceived as the temporary
best-fit for her. Therefore, it is sensible to choose the correlation coefficient as
a similarity measurement between plans and the task is to maximizes the fol-
lowing so called ”cumulative reward”: Reward
(2)
T
=
P
T
t=1
(Corr(p
∗
t
, p
t
)), where
Corr(p
∗
t
, p
t
) is the correlation value between two vectors p
∗
t
and p
t
.
Possible correlation values can be Pearson correlation or Kendall correlation.
Since the actual demand of new users is not available at the time when they
enter, it is fair to treat all the features equally in the correlation calculation.
While solving this problem, we try to recommend a rate-plan that is sufficiently
good in terms the features and that the user will accept. Thus, classic precision
measurements are not applicable. We, therefore, define the Average-Feature Pre-
diction (AFP) as a new evaluation measurement of how much of the features of
the rate-plan chosen by T new users are predictable on average:
AF P
T
=
1
T
Reward
(2)
T
4 Bandit Algorithms for the CSAR Problem
Based on the idea of the multi-armed bandit [5,6,7], in the following we translate
the new CSAR problem into a bandit problem.
Let us consider a set P of n available rate-plans to recommend to T com-
pletely new users. Each plan is associated with an unknown distribution of being
selected by users. The game of the recommender system is to repeatedly pick
up one of the rate-plans and suggest to a new user whenever she enters the
system. The ultimate goal is to maximize the cumulative reward. As defined in
previous section, the reward for our recommender system is the similarity value
similarity(p
t
, p
∗
t
). Note that the setting in present context is slightly different
from traditional MABs. In a traditional MAB only the reward of the selected
arm is revealed. In our case all the non-selected arms also get rewards after the
recommendation is made. In fact, in the case of using the indicator function,
then the non-selected rate-plans by users will get a zero reward. In the case of
using the correlation value, the rewards of the non-selected rate-plans will be the
correlation value between the two vectors p and p
∗
. However, since the distri-
butions of the rate-plans being selected are still unknown, the idea of using the
MAB algorithms for the CSAR problem is still valid. The following three MAB
algorithms are being used:
-greedy [7] aims at picking up the rate-plan that is currently considered the
best (i.e. the rate-plan that has the maximal average reward) with probability
(exploit current knowledge), and pick it up uniformly at random with probability
1 − (explore to improve knowledge). Typically, is varied along time so that
the plans get greedier and greedier as knowledge is gathered.