1
Efficient Structured Policies for Admission Control in
Heterogeneous Wireless Networks
Amin Farbod and Ben Liang
Department of Electrical and Computer Engineering
University of Toronto
Toronto, Ontario, CANADA
Email: {afarbod,liang}@comm.utoronto.ca
Abstract—In the near future, demand for Heterogeneous
Wireless Networking (HWN) is expected to to increase. QoS
provisioning in these networks is a challenging issue considering
the diversity in wireless networking technologies and the existence
of mobile users with different communication requirements. In
HWNs with their increased complexity, “the curse of dimension-
ality” problem makes it impractical to directly apply the decision
theoretic optimal control methods that are previously used in
homogeneous wireless networks to achieve desired QoS levels.
In this paper, optimal call admission control policies for HWNs
are considered. A decision theoretic framework for the problem
is derived by a dynamic programming formulation. We prove
that for a two-tier wireless network architecture, the optimal
policy has a two-dimensional threshold structure. Further, this
structural result is used to design two computationally efficient
algorithms, Structured Value Iteration and Structured Update
Value Iteration. These algorithms can be used to determine the
optimal policy in terms of thresholds. Although the first one
is closer in its operation to the conventional Value Iteration
algorithm, the second one has a significantly lower complexity.
Extensive numerical observations suggest that, for all practical
parameter sets, the algorithms always converge to the overall
optimal policy. Further, the numerical results show that the
proposed algorithms are efficient in terms of time-complexity
and in achieving the optimal performance.
Index Terms—Stochastic optimal control, quality of service,
markov processes.
I. INTRODUCTION
Heterogeneous Wireless Networking (HWN) is a major
next-generation networking architecture to support ubiquitous
wireless communications [1]. Current wireless communication
technologies can generally be classified into two groups:
local and global. Local services provide high-bandwidth and
low latency communication services over a small area, while
global services provide lower data rates to a wider area [2].
No single wireless communication technology is capable of
simultaneously providing high bandwidth to a large number of
mobile users over a wide area. HWN is a wireless networking
paradigm to overcome this limitation. Such networks consist
of several layers of different overlapping wireless networking
technologies such as WiMAX/WiFi integration.
Corresponding Author: Ben Liang, Department of Electrical and Computer
Engineering,University of Toronto,10 King’s College Road, Toronto, Ontario,
M4S 3G4, Email: liang@comm.utoronto.ca, Tel:+1-416-946-8614, Fax: +1-
416-978-4425
This work was supported in part by a grant from LG Electronics and by
Bell Canada through its Bell University Laboratories R&D program.
QoS provisioning in HWNs is a challenging issue con-
sidering the diversity of wireless networking technologies.
Conventionally, call admission control (CAC) schemes are
used in wireless networks to achieve a desired QoS level.
A CAC algorithm decides to accept or reject call or handoff
requests or to reserve resources in a resource-sharing systems.
CAC schemes for homogeneous cellular networks have been
extensively studied. These schemes can be classified into
near-optimal heuristics [3] [4] and decision-theoretic optimal
methods [5]–[7]. Furthermore, Dynamic Programming (DP)
and Markov Decision Processes (MDP) [8] are used in the
design of optimal CAC algorithms.
However, for almost all realistic modelings of networking
systems, the computational load of finding an optimal policy
by MDP algorithms is very high. Also, the size of state
space grows exponentially with system capacity. Numerical
methods [8] to solve MDP problems are iterative and as
reported in [9], there is no known strongly-polynomial time
algorithm to solve them. This can hinder the application of
optimal CAC schemes in practical scenarios. As a remedy,
one common modeling approach is to isolate one cell from
the rest of the network to avoid excessive complexity in state
space [10].
A more effective use of DP-based methods is to obtain
structural results for optimal control problems [11]–[15]. In
structural results, a DP formulation is used to characterize
the structure of possible optimal policies. Then, knowledge of
the policy structure can be exploited to design very efficient
numerical methods to find the optimal policy. As an example,
in [5], it is shown that the optimal control policy for a single
cellular Base-Station (BS) is the well-known guard-channel
policy. Then, knowing that the guard-channel policy is fully
determined by a single threshold, the authors of [5] propose an
efficient method to find it. It has been shown in the literature
that for a large class of optimal control problems the optimal
policy is threshold-based [5] [15].
To the best of our knowledge, there is no existing study
on optimal CAC schemes for heterogeneous networks. Due
to the increased complexity in HWN, direct application of
MDP algorithms is impractical. In this paper, optimal CAC
policies for HWNs are considered and some structural results
are presented. We base our algorithms on theories in optimal
control where dynamic programming methods are used to find
the optimal policy to control a random process over time
2
Underlay BS
00000
00000
00000
00000
00000
00000
00000
00000
00000
00000
00000
00000
11111
11111
11111
11111
11111
11111
11111
11111
11111
11111
11111
11111
Overlay BS
BS Coverage Area
Cluster
λ
c
λ
w
λ
in
hcc
λ
out
hcc
η
hcw
η
hwc
Fig. 1. Cluster traffic arrival and departure.
to achieve a certain optimization goal. A decision theoretic
framework for the problem is derived by dynamic program-
ming formulation. In this paper, we limit our focus to a two-
tier wireless network architecture. With some modifications,
this model can be applied to more complex scenarios. We
prove that for this architecture the optimal policy has a two-
dimensional threshold structure. Further, this structural result
is used to design two computationally efficient algorithms,
Structured Value Iteration (SVI) and Structured Update Value
Iteration (SUVI). These algorithms can be used to determine
the optimal policy in terms of thresholds. Although the first
one is closer in its operation to the conventional Value Iteration
(VI) algorithm, the second one has a significantly lower
complexity. Extensive numerical observations suggest that, for
all practical parameter sets, the algorithms always converge to
the overall optimal policy.
The rest of the paper is organized as follows. In Section II,
the system model and assumptions are presented. Section III
presents the structural results and discussion on the complexity
of algorithms to solve optimal CAC problems. In Section IV,
the proposed algorithms are explained, and numerical results
are given in section V, followed by concluding remarks in
Section VI.
II. NETWORK MODEL
A HWN can possibly have a complex configuration, in-
volving many different wireless service layers. It is generally
difficult to analytically tract such complicated scenarios to
provide insight into the optimal control of resources in HWNs.
In this work, we assume a two-tier heterogeneous wireless
network architecture consisting of an overlay and an underlay.
This basic 2-tier entity will be called a Cluster. An example
cluster is shown in Fig. 1. There are also neighboring clusters
from which horizontal handovers are possible to this cluster.
We assume tight coupling between different layers of wireless
network [16] in a cluster. In the tight coupling architecture, the
management of different layers is centralized. In what follows,
we assume that there exists a control unit which makes the
CAC decision for the underlay and overlay BSs, and that
clusters act independently and can measure the rate of external
arrival processes such as hand-overs from neighbor clusters.
Note that our mathematical analysis and control algorithms
are independent of underlying wireless technologies as long
as they satisfy some general technical requirements. However,
in the simulation section parameters are chosen with respect
to IEEE 802.16 WiMAX and IEEE 802.11 WLAN standards.
Service requests (more specifically calls in this work) arrive
according to a memoryless Poisson process, and also service
times are memoryless. Average service times are µ
c
and
µ
w
for calls inside overlay and underlay. We also assume a
memoryless mobility pattern where calls move to neighbor
clusters or different layers at exponentially distributed times
with rates given in Table I. It is clear that these assumptions
result in exponential channel holding times [17]. This is an
essential requirement in the application of MDP methods. In
this paper, we focus on call-level QoS, which is common in
CAC literature. Further, the fixed channel allocation (FCA)
scheme is used and C
c
and C
w
denote the capacity of overlay
and underlay in terms of the maximum number of calls they
can accommodate. FCA easily applies to various wireless
technologies with channel being frequency, time-slot or code
assignment. Based on the results in [18] and [19], we consider
the case where the number of available voice/multimedia
channels in underlay/overlay can be quantized.
In our event-based DP, we associate costs to undesirable
control decision events. These costs correspond to the drop-
ping or blocking of arriving calls, and they are incurred when
a call admission request is rejected by the cluster control
unit. They reflect the degradation in the QoS from the service
provider’s perspective or the inconvenience of service denial
perceived by users. The call and handoff arrival rates and their
corresponding rejection costs are shown in Table I. Throughout
the rest of this paper, every call type is called a class.
In the study of CAC schemes several optimality criteria are
considered. The most common ones are minimization of a
total cost (objective) function and minimization of the blocking
probability given some hard constraints on dropping probabili-
ties. In [5], these are refereed to as MINOBJ and MINBLOCK,
respectively. The main advantage of MINBLOCK lies in
the fact that it can guarantee some upper bounds on the
dropping probabilities. This can also be achieved by MINOBJ
by adjusting cost ratios. Furthermore, MINBLOCK has the
drawback of not taking into account how much resource is
wasted in reservation to achieve those bounds [10]. In this
paper, we focus on MINOBJ for its flexibility. We can formally
define MINOBJ as
MINOBJ : min g
π
=
P
L
k=1
C
(k)
R
λ
k
P
(k)
B
(1)
where C
(k)
R
is the cost of rejecting a call request of class k,
λ
k
is the arrival rate of class k calls, P
(k)
B
is the blocking
(dropping) probability for that class and L is the total number
of call classes.
III. OPTIMAL CAC POLICY
Decision theoretic optimization for Markovian processes is
a well-known stochastic control method [20]. The Markov
property allows significant reduction in tabular programming
complexity and in some cases makes it possible to obtain
structural results. An MDP is determined by four components:
3
# Rate Rejection Cost Description
1 λ
c
C
N BC
New calls to Overlay
2 λ
w
C
N BW
New calls to Underlay
3 η
hcc
C
HDC C
Handoff to Overlay from Overlay
4 η
hcw
C
HDC W
Handoff to Overlay from Underlay
5 η
hwc
C
HDW C
Handoff to Underlay from Overlay
TABLE I
CALL ARRIVAL/HANDOFF RATES AND REJECTION COSTS.
state space S, action space A, state transition probabilities
P ( .), and a cost function C(.). The performance criteria can
be formulated with respect to finite or infinite horizons and
for average-cost or discounted-cost problems. The solution to
an MDP is called a policy or rule. A policy maps the state
space to actions π : S → A, such that the optimization goal
is achieved. A large class of policies, in which the decision is
independent of time, are called stationary policies.
In this work, we wish to minimize the average expected cost
for an infinite-horizon problem. This reflects our concern about
long-run QoS performance. We start with a finite-horizon
optimal cost function and we show that the solution to the
infinite-horizon problem has the same structure. Let us denote
by V
k
(i, j) the optimal cost function for a k-stage problem
with the initial state (i, j) where i is the number of calls in
overlay and j is the number of calls in underlay at the start of
the decision epoch. Using the uniformization technique [21],
we can write V
k+1
(i, j) recursively as
V
k+1
(i, j) =
λ
c
v
max
min(V
k
(i, j) + C
NBC
, V
k
(i + 1, j))
+
λ
w
v
max
min(V
k
(i, j) + C
NBW
, V
k
(i, j + 1))
+
λ
in
hcc
v
max
min(V
k
(i, j) + C
HD CC
, V
k
(i + 1, j))
+
iη
hcw
v
max
min(V
k
(i − 1, j) + C
HD CW
, V
k
(i − 1, j + 1))
+
jη
hwc
v
max
min(V
k
(i, j − 1) + C
HD W C
, V
k
(i + 1, j − 1))
+
iµ
c
v
max
V
k
(i − 1, j) +
jµ
w
v
max
V
k
(i, j − 1)
+
λ
out
hcc
v
max
V
k
(i − 1, j) + (1 −
v
out
(i, j)
v
max
)V
k
(i, j) (2)
where v
out
(i, j) is the rate of going out of state s = (i, j),
v
out
(i, j)=λ
c
+ λ
w
+ λ
in
hcc
+ λ
out
hcc
+ iη
hcw
+ jη
hwc
+ iµ
c
+ jµ
w
, (3)
λ
out
hcc
= iη
hcc
, and v
max
is the uniformization parameter such
that v
max
≥ v
out
(i, j) for every (i, j) pair. Since v
out
(i, j) is
increasing in i and j, we choose v
max
= v
out
(C
c
, C
w
). Equa-
tion (2) consists of nine terms, each reflecting one possible
event; the first three terms reflect arrivals to the cluster, the
fourth and fifth terms account for vertical handovers, the next
three terms are for departure events and the last term is due to
the uniformization technique where staying in the same state
is possible. We also assume the following boundary conditions
V
k
(C
c
+ 1, j) = ∞ and V
k
(−1, j) = 00 ≤ j ≤ C
w
V
k
(i, C
w
+ 1) = ∞ and V
k
(−1, j) = 00 ≤ i ≤ C
c
. (4)
A. Optimality of Threshold-Based Policy
We show that the optimal policy to minimize the average
cost for the system model given in Section II is a 2D threshold-
based policy. In a single threshold system, that threshold is
independent of the system state. When the system state is more
complex, such as in the HWNs case, the threshold for the
operation of one system component might depend on the state
of another one. In our scenario, it gives rise to a 2D threshold
structure.
From (2), it can be seen that when a call of class L arrives,
it is only admitted if V
k
(i
0
, j
0
) − V
k
(i, j) ≤ C
L
R
, where state
s = (i, j) is the current state, state t = (i
0
, j
0
) is the next state
if we admit the call, and C
L
R
is the rejection cost for class L.
Let us define two difference operators for V
k
(i, j),
∆
i
V
k
(i, j) = V
k
(i, j) − V
k
(i − 1, j)
∆
j
V
k
(i, j) = V
k
(i, j) − V
k
(i, j − 1). (5)
For every fixed j there is a sequence of ∆
i
V
k
(i, j) for i =
1 . . . C
c
, and vice versa. In what follows we claim that the
sequences of ∆
i
V
k
(i, j) and ∆
j
V
k
(i, j) are increasing in i
and j, respectively. For the proof refer to the Appendix.
Lemma 1: V
k
(i, j) is convex and monotonically non-
decreasing in i (or j) for every fixed j (or i).
It has been shown in [21] that for average-cost problems
with finite S and A, the optimal policy is stationary. Further,
we are only interested in stationary policies which result in
irreducible chains. The chain defined by V
k
(i, j) is also aperi-
odic since it contains loops into the same state. According to
Theorem (6.6.2) in [21], for irreducible and aperiodic markov
decision processes the difference of upper and lower bounds
of V
k+1
(i, j) − V
k
(i, j) converges to the optimal average cost
per unit time when k → ∞. Also, Theorem (6.6.1) in [21]
implies that the optimal per-unit-time average cost function has
the same structure as V
k
(i, j) defined in (2). Hence, structural
results on V
k
(i, j) would directly hold for the infinite-horizon
per-unit-time cost function.
Theorem 1: A 2D threshold-based policy is an optimal
solution to the control problem with the system model given
in (2).
Proof: Without loss of generality, let us assume that a
call of class L arrives to overlay when the system state is
s = (i − 1, j
0
). The proof for arrivals to underlay is similar.
If the call is admitted, increase in the optimal cost function is
∆
i
V
k
(i, j
0
). We show that the CAC decision can be expressed
in terms of thresholds determined by ∆
i
V
k
(i, j
0
) and C
L
R
.
From Lemma 1, we know that the sequence of ∆
i
V
k
(i, j
0
)
is increasing in i. If there is an
ˆ
i for which ∆
i
V
k
(
ˆ
i, j
0
) ≤ C
L
R
and ∆
i
V
k
(
ˆ
i+1, j
0
) > C
L
R
, then
ˆ
i is the threshold for admission
to overlay when there are j
0
calls in underlay. Otherwise, if
for every
ˆ
i ∈ {1, . . . , C
c
} we have that ∆
i
V
k
(
ˆ
i, j
0
) ≤ C
L
R
then that call is of high priority and it is only rejected when
the system is full. Also, if for every
ˆ
i, ∆V
j
0
(
ˆ
i, j) > C
L
R
then
4
that call class is of low priority and it is never admitted to the
system.
Note that for every call class of L, threshold
ˆ
i depends on
∆
i
V
k
(
ˆ
i, j
0
) which in turn depends on j
0
. This implies that the
threshold for overlay operation depends on the underlay state.
Therefore, the optimal control policy has to be 2D threshold-
based to account for this correlation.
B. CAC Algorithm
We denote by π = hTh
c
[C
w
, M], Th
w
[C
c
, N]i the class
of threshold-based polices. Here, M is the number of call
classes entering overlay and N is the number of call classes
entering underlay. Every class within M or N would be called
a subclass. In our scenario M is 3 and N is 2. The CAC
algorithm when system state is s = (i, j) at the arrival epoch
and policy π is employed is given in Algorithm 1. When a
call of subclass L
0
arrives to overlay (underlay), it is only
admitted if the number of active calls in overlay (underlay)
is less than the threshold for that call-type. This threshold is
a function of call subclass and number of calls in the other
network underlay (overlay).
A CAC algorithm is fully determined given policy π in
terms of thresholds. However, finding these values is a non-
trivial problem. Efficient computation of these thresholds is
considered in the next section.
Algorithm 1 2D Threshold-Based CAC
Input: π = hTh
c
[C
w
, M], Th
w
[C
c
, N]i
A Call of class L arrives
It belongs to subclass L
0
Output: Admission Decision
1: if Arrival to overlay then
2: if i < Th
c
(j, L
0
) then
3: return Admit
4: else
5: return Reject
6: end if
7: else {Arrival to underlay}
8: if j < Th
w
(i, L
0
) then
9: return Admit
10: else
11: return Reject
12: end if
13: end if
C. Finding Policy π
A major requirement for CAC algorithms is their adaptivity
to network traffic dynamics. Since this is generally achieved
by periodically updating the admission policy, the algorithm
computational load has to be minimal. Depending on the
system size, the computation cost of solving a general MDP
can be very high. Several methods such as Value Iteration (VI),
Policy Iteration (PI) and Linear Programming (LP) methods
are developed to solve general MDP problems [8].
According to [9], no strongly-polynomial algorithm is
known for solving MDPs. Although MDPs can be solved by
conversion to LP problems, polynomial-time algorithms for
LP are inefficient and impractical. On the other hand, practical
LP algorithms can result in exponential time-complexity in the
worst case when used to solve MDPs. Consequently, there are
no efficient and practical polynomial-time algorithms to solve
MDPs. Therefore, the computation cost of finding thresholds
for the optimal policy can be a burden if we use any of
these techniques. However, when we know about the optimal
solution structure, we might be able to exploit this knowledge
to solve the problem more efficiently.
Generally, either direct or indirect methods can be employed
to find the CAC parameters, i.e., policy thresholds. Direct
methods require calculating the average cost for a given
policy π
1
. This can be done by modeling the system as
a continuous markov chain (CTMC). Note that every MDP
problem given a policy π
1
can be analyzed as a Markov
chain. Then Gaussian elimination-like methods can be used
to find state probabilities and to calculate the average cost.
Once we have the average cost we can use methods such as
multidimensional bisection search [22] to find the parameters
that minimize it. The problem with this method is that for a
two-tier network each having capacity n, the size of Markov
chain state space would be O(n
2
) and Gaussian elimination
would take O((n
2
)
3
) = O(n
6
).
However, as explained previously, CAC algorithms have to
be light weight to be of any practical use. In indirect methods
we avoid a direct evaluation of cost function. Instead we use
an iterative approximation. Along with that, we use our prior
knowledge of optimal policy structure to further improve the
algorithm time-complexity.
IV. EFFICIENT COMPUTATIONAL ALGORITHMS
In this section, we introduce efficient computational algo-
rithms to find the optimal CAC policy. We first describe the
conventional Value Iteration (VI) algorithm. We then propose
two efficient algorithms called Structured Value Iteration (SVI)
and Structured Update Value Iteration (SUVI). The basic
principle of these algorithms is similar to VI. However, we
use our prior knowledge of the optimal solution structure to
eliminate unnecessary computations.
A. Conventional VI Algorithm
Conventional Value Iteration (VI) algorithm is based on the
Bellman-Ford iterative equation [8],
V
n
(s) = min
a∈A(s)
{c
s
(a) +
X
t∈S
P
st
(a)V
n−1
(t)}. (6)
Note that this equation is backward in time, such that V
0
(.) is
the cost at the end of the process. In every iteration V
n
(s) is
calculated for ∀s ∈ S. Here, S is the state space, and A(s) is
the set of possible actions at state s. P
st
(a) is the transition
probability of going form s to t having taken action a, and
c
s
(a) is the cost of taking action a in state s. In our model,
the system state has two components, the number of calls in
overlay i and the number of calls in underlay j; s = (i, j). For
every incoming call, either new or hand-off, at any state two
5
actions are possible: accept (denoted by 1) or reject (denoted
by 0); A(s) = {0, 1}.
To find the state transition probabilities P
st
(a), we use
fictitious decision epochs [21]. The computational load of
evaluating the cost function in every step highly depends
on the density of the P
st
(a) matrix. When times between
decision epochs are exponentially distributed we can reduce
the computation cost by introducing fictitious decision epochs
at which no real decision has to be made. These correspond to
departure events when no action is taken. By this technique,
at every decision epoch either real or fictitious, the system
state can only change to adjacent states, making many terms
in P
st
(a) zero. However, to keep track of the epoch type
we have to extend the state space by one dimension. The
increased computation cost due to this enlarged state-space
is compensated by the reduction in the P
st
(a) density.
We define the new state variable to be a triple s = (i, j, k).
Here, k is the departure or arrival type. We already have 5
call types from Table I. We add a fictitious call event type
of 0 which corresponds to call departures with a fictitious
decision of a = 0 to be taken at departure events. In addition,
since the decision epochs can be at any randomly distributed
time, a Semi-Markov Decision Process (SMDP) model need
to be used [21]. Again, we take the uniformization rate to be
v
max
= v
out
(C
c
, C
w
). Also, we have to determine v
s
(a), the
rate of going out of state s having taken action a. Here we
give the transition probabilities for some of the state-action
combinations in terms of transition rates with P
st
(a) =
q
st
(a)
v
s
(a)
and s = (i, j, k):
q
st
(a = 1) =
(i + 1)( η
hcc
+ µ
c
) t = (i, j, 0)
jµ
w
t = (i + 1, j − 1, 0)
λ
c
t = (i + 1, j, 1)
λ
w
t = (i + 1, j, 2)
λ
in
hcc
t = (i + 1, j, 3)
(i + 1)η
hcw
t = (i + 1, j, 4)
jη
hwc
t = (i + 1, j, 5)
(7)
for k ∈ {1, 3} and v
s
(a = 1) = v
out
(i + 1, j) and v
out
(i, j)
given in (3). Another example for k = 4 is
q
st
(a = 0) =
(i − 1)( η
hcc
+ µ
c
) t = (i − 2, j, 0)
jµ
w
t = (i − 1, j − 1, 0)
λ
c
t = (i − 1, j, 1)
λ
w
t = (i − 1, j, 2)
λ
in
hcc
t = (i − 1, j, 3)
(i − 1)η
hcw
t = (i − 1, j, 4)
jη
hwc
t = (i − 1, j, 5)
(8)
with v
s
(a) = v
out
(i − 1, j). Note that in the above, a hand-off
request from overlay to underlay was initially rejected (a = 0)
leaving only i − 1 calls in overlay at the start of the decision
epoch. We specify the boundary conditions by defining
V
n
(C
c
+ 1, j, k) = ∞ for all j and k
V
n
(i, C
w
+ 1, k ) = ∞ for all i and k
V
n
(−1, j, k) = 0 for all j and k
V
n
(i, −1, k) = 0 for all i and k (9)
0 5 10 15 20 25 30
0
2
4
6
8
10
12
14
16
18
20
CELL State i
WLAN State j
Admission Region
Rejection Region
Border Region
Fig. 2. SVI algorithm operation; AR, BR and RR for calls coming to underlay
and D = 1.
For SMDP, (6) needs to be modified to reflect the semi-Markov
state transition rates. We define operator T
V
[V (.), s, a] as
T
V
[V (.), s, a]=c
s
(a)v
s
(a)
+
v
s
(a)
v
max
X
t∈S
P
st
(a)V (t)
+
µ
1 −
v
s
(a)
v
max
¶
V (s). (10)
Given this operator we can rewrite (6) for SMDPs as
V
n
(s)= min
a∈A(s)
{T
V
[V
n−1
, s, a]}. (11)
B. SVI Algorithm
Theorem 1 states that the optimal solution is a 2D threshold
policy, implying that the admission region for any call type
should be a closed area. An example of this is shown in Fig. 2.
For any given policy π
1
and call subclass, we can partition
the state space into three disjoint areas, called Accept-Region
(AR), Border-Region (BR), and Reject-Region (RR). We define
the region indicator function I
R
(s, p) for state s = (i, j) and
call request of subclass p as
I
R
(s, p) =
AR i − Th
c
(j, p) < −D
BR |i − Th
c
(j, p)| ≤ D
RR i − Th
c
(j, p) > D.
(12)
If a state is within distance D of the threshold level then
it is in BR. D acts as a tuning parameter, determining the
size of area we are willing to re-evaluate in every iteration.
An example of I
R
(s, p) classification is shown in Fig. 2 for
D = 1, where dotted states correspond to the threshold levels.
The indicator function for the underlay subclasses is similar.
Given the indicator function I
R
(s, p), we can redefine the
action space A(s) as A
0
(s),
A
0
(s) =
{0} if I
R
(s, p) = RR
{1} if I
R
(s, p) = AR
{0, 1} if I
R
(s, p) = BR.
(13)
Here, we are limiting the set of possible actions. The idea
is that for states inside the admission region it would be
unnecessary to consider a possible reject action if they are
not close to the border. Note that the cost function evaluation
