1
Domain of Competence of XCS
Classifier System
in Complexity Measurement Space
Ester Bernad´o Mansilla
and Tin Kam Ho
Computer Engineering Department
Enginyeria i Arquitectura La Salle, Ramon Llull University
Quatre Camins, 2. 08022 Barcelona, Spain
E-mail: esterb@salleurl.edu Tel: +34 932 902 433
Computing Sciences Research Center, Bell Laboratories, Lucent Technologies
700 Mountain Avenue, 2C-425
Murray Hill, NJ 07974-0636 USA
E-mail: tkh@research.bell-labs.com Tel: +1 908 582 5989
Abstract
The XCS classifier system has recently shown a high degree of competence on a variety of data mining
problems. But to what kind of problems XCS is well and poorly suited is seldom understood, especially
for real-world classification problems. The major inconvenience has been attributed to the difficulty of
determining the intrinsic characteristics of real-world classification problems. This paper investigates the
domain of competence of XCS by means of a methodology that characterizes the complexity of a classi-
fication problem by a set of geometrical descriptors. In a study of 392 classification problems along with
their complexity characterization, we are able to identify difficult and easy domains for XCS. We focus
on XCS with hyperrectangle codification, which has been predominantly used for real-attributed domains.
The results show high correlations between XCS’s performance and measures of length of class boundaries,
compactness of classes and non-linearities of decision boundaries. We also compare the relative performance
of XCS with other traditional classifier schemes. Besides confirming the high degree of competence of XCS
in these problems, we are able to relate the behavior of the different classifier schemes to the geometrical
complexity of the problem. Moreover, the results highlight certain regions of the complexity measurement
space where a classifier scheme excels, establishing a first step towards determining the best classifier
scheme for a given classification problem.
2
Index Terms
Learning classifier systems, geometrical complexity, genetic algorithms, pattern recognition, machine
learning, classification.
I. INTRODUCTION
XCS [1], [2] is a classifier system that combines reinforcement learning [3] and genetic algorithms
(GA) [4], [5] to evolve a set of rules representing the target concept. XCS descends from the lineage of
learning classifier systems (LCS), which were first introduced by Holland [4], [6], [7]. Its success and
robust performance in a variety of domains establish XCS as one of the major developments of learning
classifier systems.
Recent investigations of XCS have been focused on a variety of aspects, with the common goal of
improving our understanding of the system. The result has often been better performance and wider
applicability in several domains. Some of these studies investigate XCS from an algorithmic point of
view. Some remarkable efforts in this direction are [8], [9]. They investigate the dynamics of the different
components of XCS, and how these components interact to evolve optimal rules that represent the target
concept in an accurate and compact way. Other studies focus on certain components of the algorithm,
such as the deletion algorithms [10], the definition of fitness [11], or the knowledge representation [12]–
[14]. Most of these studies are restricted to artificially designed problems, because they are easy to analyze
and allow control of their degree of complexity to some extent.
Other types of investigations focus on the applicability of XCS to real-world domains. In [15], XCS is
applied to a data mining problem with a high degree of performance, both in terms of accuracy rate
and explanatory capabilities. This study is extended in [16] with a varied set of data mining problems,
where XCS appears to perform competitively with respect to other classifier techniques, such as nearest
neighbors and decision trees. These studies provide a comparison of accuracy rates between XCS and
other classifier schemes
1
and show that XCS is competitive. However, this information is of limited use,
as there is a lack of a deeper understanding of the general behavior of XCS in real-world problems,
i.e., what types of problems XCS is well or poorly suited to, and the reasons of the success or failure
of XCS compared to other classifier schemes. Moreover, it is well known that there is not any classifier
scheme that globally dominates in every domain. Instead, there might be some types of problems where
a particular classifier scheme excels [17]. Therefore, a major issue of current research is to identify the
domains of competence of a particular classifier scheme. One of the major obstacles in this kind of
1
In the context of XCS, a classifier is a rule and a set of associated parameters. XCS evolves a set of such classifiers. The term
classifier is also used by the pattern recognition community to refer to the whole system that classifies (e.g., nearest neighbor
classifier, linear classifier, etc). In section II we use this term following the terminology used in XCS. In the rest of the paper the
term classifier is used in the sense of the whole system, unless properly indicated.
3
investigation is the difficulty of characterizing the differences between various real-world problems, and
relating the classifier’s behavior to such differences.
This paper is in line of recent efforts in the pattern recognition community to characterize the behavior
of a classifier system related to the features of the problem. We identify the features of a problem that
are most relevant to classification accuracy as measurements of the problem’s geometrical complexity.
The paper analyzes the domain of applicability of XCS in such a measurement space. In particular, the
paper addresses the following issues:
1) What is the complexity of a real-world classification problem? We will analyze the sources of difficulty
in a classification problem. We will focus on the description of the geometrical complexity of the
problem, such as the degree of clustering of the points of the same class, the proximity between the
classes, and other factors that are critical for classification accuracy. We emphasize that we consider
only those descriptors that can be extracted directly from the dataset. Therefore the analysis can be
applied to arbitrary real-world data without reliance on an assumption of the generating model.
2) Given a real-world classification problem characterized by a set of complexity descriptors, is XCS well suited
for that problem? That is, given a certain classification problem, is XCS applicable? Will XCS be
able to extract an accurate knowledge representation? This issue has special relevance when we
apply XCS to a real-world classification problem, where the intrinsic difficulties of the problem are
often unknown, and are difficult to separate from the inadequacy of the classification algorithm.
With the characterization of a real-world classification problem by a set of complexity descriptors
that are not directly dependent on the classifier, we are able to investigate the relation between
XCS’s performance and problem complexity. This study will identify which kind of problems XCS
is particularly well suited and poorly suited to. In order to infer conclusions which show the general
tendency of XCS in real-world problems, we will use an extended set of 392 problems. This gives
better coverage on the variety of problems than some previous studies on XCS’s performance [16]
that relied on a small set of problems (about 20 typically) or only artificial ones [18].
3) Given a classification problem, what is the best suited classifier scheme? As observed previously, there
is not an outstanding classifier scheme that dominates in all sorts of classification problems. There
are certain types of classification problems for which particular kinds of classifiers are best suited.
Thus, a central issue is to identify the good matches between classifier schemes and problems. We
will analyze XCS’s performance in comparison to other learning algorithms, and relate the results to
the complexity of the problem. Previous studies could hardly explain why XCS was better or worse
than a particular classifier, and in which cases this happened. Relating the respective performances
to the complexity characterization of each problem will give us a handle in understanding what
kind of problems are best suited for a particular classifier and, even more important, why.
This study is aimed at a better understanding of XCS’s behavior in real-world classification problems.
4
Detecting where XCS has difficulties may lead to improvement of the method. The study will also lay
the bases for the characterization of the problems in a space of complexity metrics, and the identification
of what kind of classifiers are more appropriate for certain regions of the measurement space.
The rest of this paper is structured as follows. First, we present a brief overview of XCS, describing
how the different components are designed to achieve the learning goals. Next, since XCS’s performance
depends both on the algorithmic components and the knowledge representation, we analyze the knowl-
edge representation used in domains with real valued attributes. We investigate some cases where XCS
encounters different levels of difficulty due to the geometry of the problem. Section IV analyzes the
sources of difficulty of real-world classification problems, and proposes several measures that represent
different aspects of the problem complexity. Next, we characterize the behavior of XCS with respect to the
complexity of the problem, and identify regions in the measurement space where the easiest problems and
the most difficult problems for XCS are located. Section VI compares XCS with other classifier schemes,
trying to determine the domains of competence of each classifier scheme. Finally, we present our main
conclusions and discuss future work.
II. DESCRIPTION OF XCS
XCS represents the knowledge extracted from the problem in a set of rules. This ruleset is incrementally
evaluated by means of interacting with the environment, through a reinforcement learning scheme, and is
improved by a search mechanism based on a genetic algorithm. The following is a brief description of
XCS. Although XCS is applicable to single-step and multi-step problems, we restrict our description of
XCS to single-step tasks like classification problems within the scope of this paper. For more details, the
reader is referred to [1] and [2] for an introduction of XCS, and [19] for an algorithmic description.
A. Representation
XCS evolves a population [P] of classifiers where each classifier has a rule and a set of associated
parameters estimating the quality of the rules. Each rule consists of a condition part and an action part:
. The condition specifies the set of input states where the classifier can be applied.
For binary inputs, the condition is usually represented in the ternary alphabet:
, where
is the
length of the input string. In this case, a condition
!
"#"$"$
%
matches an input example
'&
(&
!
"#"$"#)&
*%
,
if and only if
+
,-/.
&
-102-3.
. The symbol
, called don’t care, allows the formation of generalizations
in the rule’s condition. The action part of the rule specifies the action or class that the classifier proposes
when its condition is satisfied. It is coded as an integer.
Three main parameters estimate the quality of each classifier: a) the payoff prediction
4
, an estimate
of the payoff that the classifier will receive if its condition matches the input and its action is selected,
b) the prediction error
5
, which estimates the average error between the classifier’s prediction and the
received payoff and c) the fitness
6
, an estimate of the accuracy of the payoff prediction. There are other
5
parameters qualifying each classifier, such as: the experience of the classifier (denoted as exp), the average
size of the action sets where the classifier has participated (as), the time-step of the last application of
the genetic algorithm (ts) and the number of actual micro-classifiers this macroclassifier
2
represents, called
numerosity (num). These are described in the following.
B. Performance Component
At each time step, an input
&
is presented to the system. Given
&
, the system builds a match set [M],
which is formed by all the classifiers in [P] whose conditions are satisfied by the input example. If the
number of actions represented in [M] is less than a threshold
, then covering is triggered. Covering
creates new classifiers with a condition matching the current input and an action selected randomly
from those not present in [M]. From the resulting match set, an action must be selected and sent to
the environment. For this purpose, a payoff prediction
%
is computed for each action
in [M].
%
estimates the payoff that the system will receive if action
is chosen. It is computed as a fitness-weighted
average of the predictions of all classifiers proposing that action. The system chooses the winning action
based on these prediction values. The chosen action determines the action set [A] which consists of all
the classifiers in [M] advocating this action.
In classification, the winning action is usually selected using either pure explore mode or pure exploit
mode. In pure explore mode, the action is selected randomly. This makes sense during training, i.e., when
the system is learning the consequences of all possible actions for a given input. In pure exploit mode,
the action is selected deterministically according to the highest prediction. This is used in test, that is,
when the system classifies new unseen instances based on the knowledge it has acquired.
C. Reinforcement Component
Once the action is sent to the environment, the environment returns a reward
, which is used to update
the parameters of the classifiers in [A]. First, the prediction of each classifier is adjusted as follows:
4
4
4
%
(1)
where
(
) is the learning rate. Next, the prediction error is updated:
5
5
4
5
%
(2)
Then, the classifier’s accuracy is computed as an inverse function of the classifier’s error:
.
5
!
5
"
%
$#&%
5
('
5
"
otherwise
(3)
2
Classifiers in XCS are macroclassifiers, i.e., each classifier represents num micro-classifiers having identical conditions and actions
[19].
