A compiler and simulator for

partial recursive functions over neural networks

João Neto

1

,JoséFélixCosta

2

,

Paulo Carreira

3

,MiguelRosa

4

1

Departamento de Informática, Faculdade de Ciências da

Universidade de Lisboa, C5 – Piso 1, 1700 Lisboa,

PORTUGAL

jpn@di.fc.ul.pt

2

Departamento de Matemática, Instituto Superior Técnico,

Av. Rovisco Pais, 1049-001 Lisboa,

PORTUGAL

fgc@math.ist.utl.pt

3

OBLOG Software S.A., Alameda António Sérgio 7-1A

2795-023 Linda-a-Velha

PORTUGAL

pcarreira@oblog.pt

4

Fundação para a Computação Científica Nacional

Av. do Brasil, 101. 1700-066 Lisboa.

PORTUGAL

mar@fccn.pt

Abstract. In [

6

]and[

8

] it was shown that Artificial Recurrent

Neural Networks have the same computing power as Turing

machines. A Turing machine can be programmed in a proper high-

level language - the language of partial recursive functions. In this

paper we present the implementation of a compiler that directly

translates high-level Turing machine programs to Artificial

Recursive Neural Networks. The application contains a simulator

that can be used to test the resulting networks. We also argue that

these experiments provide clues to develop procedures for

automatic synthesis of Neural Networks from high-level

descriptions.

1. Introduction

The field of Artificial Recurrent Neural Networks (ARNNs) is meeting a

lot of excitement nowadays. Both because of their achievements in solving

real world problems and their simplicity of the underlying principles that

still allow them to mimic their biological counterparts. All this excitement

attracts people from many different fields such as Neurophisiology and

Computer Science.

We introduce our subject from a Computer Science perspective. The

view we are interested in, is the one in which an ARNN can be seen as a

computing mechanism able to perform some kind of computation based on

a program coded as a specific arrangement of neural artifacts, like neurons

and synapses. This work implements a compiler and a simulator based on

the previous Turing Universality of Neural Nets (Revisited) paper, [4]. In

[3], [7] and [5] similar ideas are given but they are based on higher level

languages.

We start by giving the underlying theoretical context on which it is

based. In section 2 we give a brief review of the concept of partial

recursive function. In section 3 we present our approach for constructing

neural networks from partial recursive functions. The explanation of how

we adapted the design of [4] into a compiler is given in section 4. Section

5 refers to the simulator and usage examples and section 6 concludes this

paper. The simulator is freely available at www.di.fc.ul.pt/~jpn/

netdef/nwb.html.

2. Partial Recursive Function theory

When informally speaking of a neural computer, one could be motivated

about what could it be like the language to program such a machine. The

language that we will use is the one of partial recursive functions (PRF).

Although primitive when compared to modern computer languages, it is

simple and powerful enough to program any mechanism with same

computing power as a Turing machine. Surely, building complex

programs with this language would be very difficult and more appropriate

languages exist. For our purposes however, this language is suited.

The PRF theory identifies the set of computable functions with the set

of partial recursive functions. We shall use a(x

1

,…,x

n

)

≡

b(x

1

,…,x

n

)to

denote equality of the expressions a(x

1

,…,x

n

)andb(x

1

,…,x

n

), if and only

if both a(x

1

,…,x

n

)andb(x

1

,…,x

n

) are defined for all (x

1

,…,x

n

) or both

undefined.

The axioms also called primitive functions are:

• W

that denotes the zero-ary constant 0;

•

S

that denotes the unary successor function S(x)=x+1;

• U(i,n) that for i and n fixed, 1≤i≤n, denotes the projection

function U

i,n

(x

1

, ..., x

n

)=x

i

.

The construction rules are:

•

C

,denotingcomposition.Iff

1

,…,f

k

are n-ary PRFs, and g is a

k-ary PRF, then the function h defined by composition,

h(x

1

,…,x

n

) ≡ g(f

1

(x

1

,…,x

n

), …, f

k

(x

1

,…,x

n

)), is a PRF;

• R,

denoting recursion. If f is a n-ary PRF and g is a (n+2)-ary

PRF, then the unique (n+1)-ary function h, defined by

1) h(x

1

,…, x

n

,0)≡ f(x

1

,…,x

n

)and

2) h(x

1

,…, x

n

, y+1) ≡ g(x

1

,…,x

n

,y,h(x

1

,…, x

n

,y));

is a PRF;

•

M,

denoting minimalisation.Iffisa(n+1)-aryPRF,then

h(x

1

,…, x

n

)

≡µ

y

(f(x

1

,…, x

n

,y)=0)isalsoaPRF,where

µ

y

(f(x

1

,…, x

n

,y)=0)=

least y such that f(x

1

,..,x

n

,y)=0 and

∀

z

≤

y: f(x

1

,…,x

n

,z) is defined

undefined, otherwise

For instance, f(x,y)=x+1 is a PRF and is described by the expression

C(U(1,2),S). The function f(x,y)=x+y is also a PRF described by the

expression

R(U(1,1),C(U(3,3),S

). In fact, it can be shown that

every Turing computable function is a PRF. More details on PRF theory

can be found in [1] or [2].

3. Coding PRF into ARNNs

Finding a systematic way of generating ARNNs from given descriptions of

PRF’s greatly simplifies the task of producing neural nets to perform

certain specific tasks. Furthermore, it also gives a proof that neural nets

can effectively compute all Turing computable functions as treated in [4].

In this section we briefly describe the computing rule of each processing

element, i.e., each neuron. Further we present the coding strategy of

natural numbers to load the network. Finally, we will see how to code a

PRF into an ARNN.

3.1 How do the processing elements work?

Likein[4]wemakeuseof

σ

-processors. In each instant t each neuron j

updates its activity x

j

in the following non-linear way:

x

j

(t+1) = σ(

∑

i=1

N

a

ji

x

i

(t) +

∑

k=1

M

b

jk

u

k

(t) + c

j

)

where a

ji

,b

jk

and c

j

are rational weights; N is the number of neurons, M the

number of input streams u

k

;andσ is the continuous function defined

below:

σ

(x) =

1,x

≥

1

x,0<x<1

0,x

≤

0

3.2 How can we represent the numbers ?

We use an unary representation where each natural number is represented

as a rational number by means of a mapping

α

where, for each n,

α

(n) is

given by

∑

i=0

n

10

1-i

3.3 How to construct the ARNN ?

The following three net schemata were implemented to compute the

corresponding three axioms of recursive function theory. Changes were

made with respect to [4]. First, the W axiom is provided with two

additional neurons. Second, each U(i,n) axiom is constructed with only

five neurons making it more efficient.

fig. 1 – Axioms (i) W; (ii) S; (iii) U(i,n).

0.1

OUT

IN

x+1

x

–0.9

(ii)

IN

OUT

0

0.1

(i)

...

...

x

i

x

n

x

1

x

i

OUT

IN

(iii)

The rules are illustrated by the net schemata of figures 2, 3 and 4,

where grey coloured circles represent repeated neurons. The Sg box

represents a subnet that finds if a given number is positive or zero:

fig. 2 – Composition.

fig.3–Recursion.

g(…)

OUT

h(…)

=0

IN

g

Sg

f(x

1

,…,x

n

)

OUT

f

H

–1

–1.2

10

>0

IN

y

Y

0.2

0.1

0.1

…

…

.

x

1

x

n

IN

X

1

X

n

–1

–1

X

1

X

n

–1

–1

–1

-10

–1

K

–0.9

H

OUT

–1–1

reset

Y

X

i

–1–1

reset

KH

–1

–1

x

n

……

…

.

.

.

x

1

f

1

(x

1

,…,x

n

)

h(…)

y

1

f

k

(x

1

,…,x

n

)

OUT

IN

OUT

OUT

y

k

f

1

IN

g

f

k

–1

–1

–1

–1

–1

– (k–1)

–1

–1

…

fig. 4 – Minimalisation.

fig. 5 – The signal (Sg) network.

For each PRF expression an ARNN is generated using the structure of

the expression. The construction of the ARNNs is made in a top-down

fashion, beginning with the outermost ARNN and then continuously

instantiating ARNNs until reaching the axioms.

For an expression R(U(1,1),C(S,U(3,3)) we would first build

the network for the recursion, then instantiate with the projection axiom

network and with the composition rule network, that in turn would

accommodate the successor and projection axiom networks. This

instantiation mechanism for the network schemas consists of replacing the

boxes by compatible network schemas. A box is said to be compatible

with a network schema if the number of inputs (respectively outputs) of

the box is the same as the number of inputs (respectively outputs) of the

network schema. The substitution operation of a box by a network schema

consists of connecting the box inputs (respectively outputs) to the network

inputs (respectively outputs).

4. Putting the pieces together and building the compiler

The tool we want to present to the user, should be capable of handling

more complex functions than the simple examples used in the previous

sections. Specifying a more complex function implies writing a more

complex expression. This motivates the use of a modular and yet simple

language to increase readability.

=1 if x=0

–1

x

IN

–100

2

2

100

=1 if x>0

–2

–0.1

10

0.1

0.1

=0

>0

Sg

f(x

1

,…,x

n

,y)

OUT

IN

OUT

h(...)

f

–1

–1–1

reset

Y

X

i

–0.9

Y

...

x

1

x

n

IN

X

1

X

n

–1

–1

–1

–1