Delft University of Technology
Hybrid single node genetic programming for symbolic regression
Kubalìk, Jiřì; Alibekov, Eduard; Žegklitz, Jan; Babuska, R.
DOI
10.1007/978-3-662-53525-7_4
Publication date
2016
Document Version
Accepted author manuscript
Published in
Transactions on Computational Collective Intelligence XXIV
Citation (APA)
Kubalìk, J., Alibekov, E., Žegklitz, J., & Babuska, R. (2016). Hybrid single node genetic programming for
symbolic regression. In NT. Nguyen, R. Kowalczyk, & J. Filipe (Eds.),
Transactions on Computational
Collective Intelligence XXIV
(Vol. LNCS 9770, pp. 61-82). (Lecture Notes in Computer Science (including
subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9770 LNCS).
Springer. https://doi.org/10.1007/978-3-662-53525-7_4
Important note
To cite this publication, please use the final published version (if applicable).
Please check the document version above.
Copyright
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent
of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Takedown policy
Please contact us and provide details if you believe this document breaches copyrights.
We will remove access to the work immediately and investigate your claim.
This work is downloaded from Delft University of Technology.
For technical reasons the number of authors shown on this cover page is limited to a maximum of 10.
Hybrid Single Node Genetic Programming for
Symbolic Regression
Jiˇr´ı Kubal´ık
1
, Eduard Alibekov
1,2
, Jan
ˇ
Zegklitz
1,2
, and Robert Babuˇska
1,3
1
Czech Institute of Informatics, Robotics, and Cybernetics,
CTU in Prague, Prague, Czech Republic
{kubalik,babuska}@ciirc.cvut.cz
2
Department of Cybernetics, Faculty of Electrical Engineering,
CTU in Prague, Prague, Czech Republic
3
Delft Center for Systems and Control,
Delft University of Technology, Delft, the Netherlands
Abstract. This paper presents a first step of our research on designing
an effective and efficient GP-based method for symbolic regression. First,
we propose three extensions of the standard Single Node GP, namely (1)
a selection strategy for choosing nodes to be mutated based on depth and
performance of the nodes, (2) operators for placing a compact version
of the best-performing graph to the beginning and to the end of the
population, respectively, and (3) a local search strategy with multiple
mutations applied in each iteration. All the proposed modifications have
been experimentally evaluated on five symbolic regression benchmarks
and compared with standard GP and SNGP. The achieved results are
promising showing the potential of the proposed modifications to improve
the performance of the SNGP algorithm. We then propose two variants of
hybrid SNGP utilizing a linear regression technique, LASSO, to improve
its performance. The proposed algorithms have been compared to the
state-of-the-art symbolic regression methods that also make use of the
linear regression techniques on four real-world benchmarks. The results
show the hybrid SNGP algorithms are at least competitive with or better
than the compared methods.
Keywords: Genetic Programming, Single Node Genetic Programming,
Symbolic Regression
1 Introduction
This paper presents a first step of our research on genetic programming (GP) for
the symbolic regression problem. The ultimate goal of our project is to design an
effective and efficient GP-based method for solving dynamic symbolic regression
problems where the target function evolves in time. Symbolic regression (SR) is
a type of regression analysis that searches the space of mathematical expressions
The final publication is available at link.springer.com
DOI: 10.1007/978-3-662-53525-7_4
2 Jiˇr´ı Kubal´ık, Eduard Alibekov, Jan
ˇ
Zegklitz, and Robert Babuˇska
to find the model that best fits a given dataset, both in terms of accuracy and
simplicity
4
.
Genetic programming belongs to effective and efficient methods for solving
the SR problem. Besides the standard Koza’s tree-based GP [12], many other
variants have been proposed. They include, for instance, Grammatical Evolution
(GE) [20] which evolves programs whose syntax is defined by a user-specified
grammar (usually a grammar in Backus-Naur form). Gene Expression Program-
ming (GEP) [4] is another GP variant successful in solving the SR problems. Sim-
ilarly to GE it evolves linear chromosomes that are expressed as tree structures
through a genotype-phenotype mapping. A graph-based Cartesian GP (CGP)
[18], is a GP technique that uses a very simple integer based genetic representa-
tion of a program in the form of a directed graph. In its classic form, CGP uses
a variant of a simple algorithm called (1 + λ)-Evolution Strategy with a point
mutation variation operator. When searching the space of candidate solutions,
CGP makes use of so called neutral mutations, meaning that a move to the
new state is accepted if it does not worsen the quality of the current solution.
This allows an introduction of new pieces of genetic code that can be plugged
into the functional code later on and allows for traversing plateaus of the fitness
landscape.
A Single Node GP (SNGP) [9], [10] is a rather new graph-based GP system
that evolves a population of individuals, each consisting of a single program node.
Similarly to CGP, the evolution is carried out via a hill-climbing mechanism
using a single reversible mutation operator. The first experiments with SNGP
were very promising as they showed that SNGP significantly outperforms the
standard GP on various problems including the SR problem. In this work we take
the standard SNGP as the baseline approach and propose several modifications
to further improve its performance.
The goals of this work are twofold. The first goal is to verify performance
of the vanilla SNGP compared to the standard GP on various SR benchmarks
and to investigate the impact of the following three design aspects of the SNGP
algorithm:
– a strategy to select the nodes to be mutated,
– a strategy according to which the nodes of the best-performing expression
are treated in the population,
– and a type of the search strategy used to guide the optimization process.
The second goal is to propose a hybrid variant of SNGP which incorporates
the LASSO regression technique for creating linear-in-parameters nonlinear mod-
els. We compare its performance with other state-of-the-art symbolic regression
methods which also make use of linear regression techniques.
The paper is organized as follows. Section 2 describes the SNGP algorithm.
In Section 3, three modifications of the SNGP algorithm are proposed. Exper-
imental evaluation of the modified SNGP and its comparison to the standard
SNGP and standard Koza’s GP is presented in Section 4. Section 5 describes two
4
https://en.wikipedia.org/wiki/Symbolic regression
Hybrid Single Node Genetic Programming for Symbolic Regression 3
variants of the SNGP utilizing the linear regression technique, LASSO, to im-
prove its performance. The two versions of SNGP with LASSO are compared to
other symbolic regression methods making use of the linear regression techniques
in Section 6. Finally, Section 7 concludes the paper and proposes directions for
the further research on this topic.
2 Single Node Genetic Programming
2.1 Representation
The Single Node Genetic Programming is a GP system that evolves a population
of individuals, each consisting of a single program node. The node can be either
terminal, i.e. a constant or a variable node, or a function from a set of functions
defined for the problem at hand. Importantly, individuals are not isolated in the
population, they are interlinked in a graph structure similar to that of CGP,
with population members acting as operands of other members [9].
Formally, a SNGP population is a set of N individuals M = {m
0
, m
1
, . . . , m
N −1
},
with each individual m
i
being a single node represented by the tuple m
i
=
hu
i
, f
i
, Succ
i
, P red
i
, O
i
i, where
– u
i
∈ T ∪ F is either an element chosen from a function set F or a terminal
set T defined for the problem,
– f
i
is the fitness of the individual,
– Succ
i
is a set of successors of this node, i.e. the nodes whose output serves
as the input to the node,
– P red
i
is a set of predecessors of this node, i.e. the nodes that use this indi-
vidual as an operand,
– O
i
is a vector of outputs produced by this node.
Typically, the population is partitioned so that the first N
term
nodes, at
positions 0 to N
term
−1, are terminals (variables and constants in case of the SR
problem), followed by function nodes. Importantly, a function node at position i
can use as its successor (i.e. the operand) any node that is positioned lower down
in the population relative to the node i. This means that for each s ∈ Succ
i
we
have 0 ≤ s < i [9]. Similarly, predecessors of individual i must occupy higher
positions in the population, i.e. for each p ∈ P red
i
we have i < p < N. Note
that each function node is in fact a root of a direct acyclic graph that can be
constructed by recursively traversing through successors until the leaf terminal
nodes.
2.2 Evolutionary model
In [9], a single evolutionary operator called successor mutate (smut) has been
proposed. It picks one individual of the population at random and then one of its
successors is replaced by a reference to another individual of the population mak-
ing sure that the constraint imposed on the successors is satisfied. Predecessor
4 Jiˇr´ı Kubal´ık, Eduard Alibekov, Jan
ˇ
Zegklitz, and Robert Babuˇska
lists of all affected individuals are updated accordingly. Moreover, all individuals
affected by this action must be reevaluated as well. For more details refer to [9].
The evolution is carried out via a hill-climbing mechanism using a smut
operator and an acceptance rule, which can have various forms. In [9], it was
based on fitness measurements across the whole population, rather than on sin-
gle individuals. This means that once the population has been changed by a
single application of the smut operator and all affected individuals have been
re-evaluated, the new population is accepted if and only if the sum of the fitness
values of all individuals in the population is no worse than the sum of fitness
values before the mutation. Otherwise, the modifications made by the mutation
are reversed. In [10] the acceptance rule is based only on the best fitness in the
population. The latter acceptance rule will be used in this work as well. The
reason for this choice is explained in Section 3.4.
3 Proposed Modifications
In this section, the following three modifications of the SNGP algorithm will be
proposed:
1. A selection strategy for choosing nodes to be mutated based on depth and
performance of nodes.
2. Operators for placing a compact version of the tree rooted in the best per-
forming node to the beginning and to the end of the population, respectively.
3. A local search strategy with multiple mutations applied in each iteration.
In the following text, the term ”best tree” is used to denote the tree rooted
in the best performing node.
3.1 Depthwise Selection Strategy
The first modification focuses on the strategy for selecting the nodes to be mu-
tated. In the standard SNGP, the node to be mutated is chosen at random.
This means that all function nodes have the same probability of selection ir-
respectively of (1) how well they are performing and (2) how well the trees of
which they are a part are performing. This is not in line with the evolutionary
paradigm where the well fit individuals should have higher chance to take part
in the process of an evolution of the population.
One way to narrow this situation is to select nodes according to their fitness.
However, this would prefer just the root nodes of trees with high fitness while
neglecting the nodes at the deeper levels of such well-performing trees which
themselves have rather poor fitness. In fact, imposing high selection pressure on
the root nodes might be counter-productive in the end as the mutations applied
on the root nodes are less likely to bring an improvement than mutations applied
on the deeper structures of the trees.
We propose a selection strategy that takes into account the quality of the
mutated trees, so that better performing trees are preferred, as well as the depth
![Fig. 1. Illustration of the moveLeft operator. Function nodes involved in the original and compact form of the best graph are shown in red. After the application of the moveLeft operator the population contains two occurrences of the best graph, the one represented by a sequence of nodes [0, 1, 2, 6, 8, 10] and the one represented by nodes [0, 1, 2, 3, 4, 5].](/figures/fig-1-illustration-of-the-moveleft-operator-function-nodes-1yt82o22.webp)
![Fig. 2. Illustration of the moveRight operator. Function nodes involved in the original and compact version of the best graph are shown in red. In this example, the final population contains just one occurrence of the best graph represented by a sequence of nodes [0, 7, 8, 9, 10, 11].](/figures/fig-2-illustration-of-the-moveright-operator-function-nodes-2b2n0zt2.webp)