The University of Manchester Research
High-Performance, Low-Complexity Deadlock Avoidance
for Arbitrary Topologies/Routings
DOI:
10.1145/3205289.3205307
Document Version
Accepted author manuscript
Link to publication record in Manchester Research Explorer
Citation for published version (APA):
Pascual Saiz, J., & Navaridas, J. (2018). High-Performance, Low-Complexity Deadlock Avoidance for Arbitrary
Topologies/Routings. In ACM International Conference on Supercomputing
https://doi.org/10.1145/3205289.3205307
Published in:
ACM International Conference on Supercomputing
Citing this paper
Please note that where the full-text provided on Manchester Research Explorer is the Author Accepted Manuscript
or Proof version this may differ from the final Published version. If citing, it is advised that you check and use the
publisher's definitive version.
General rights
Copyright and moral rights for the publications made accessible in the Research Explorer are retained by the
authors and/or other copyright owners and it is a condition of accessing publications that users recognise and
abide by the legal requirements associated with these rights.
Takedown policy
If you believe that this document breaches copyright please refer to the University of Manchester’s Takedown
Procedures [http://man.ac.uk/04Y6Bo] or contact uml.scholarlycommunications@manchester.ac.uk providing
relevant details, so we can investigate your claim.
Download date:09. Aug. 2022
High-Performance, Low-Complexity Deadlock
Avoidance for Arbitrary Topologies/Routings
Jose A. Pascual
The University of Manchester
Manchester, United Kingdom
jose.pascual@manchester.ac.uk
Javier Navaridas
The University of Manchester
Manchester, United Kingdom
javier.navaridas@manchester.ac.uk
ABSTRACT
Recently, the use of graph-based network topologies has been pro-
posed as an alternative to traditional networks such as tori or fat-
trees due to their very good topological characteristics. However
they pose practical implementation challenges such as the lack of
deadlock avoidance strategies. Previous proposals are either exceed-
ingly complex, underutilise network resources or lack exibility. We
propose–and prove formally–three generic, low-complexity dead-
lock avoidance mechanisms that only require local information.
The main strengths of our method are its topology- and routing-
independence and that the virtual channel count is bounded by
the length of the longest path. We evaluate our proposed mecha-
nisms against previous proposals through an extensive simulation
study to measure the impact on the performance using both syn-
thetic and realistic trac. First we compare against a well-known
HPC mechanism for dragony and achieved similar performance
level. Then we moved to Graph-based networks and show that
our mechanisms can greatly outperform traditional, spanning-tree
based mechanisms, even if these use a much larger number of vir-
tual channels. Overall, we nd that our proposal provides a simple,
exible and high performance deadlock-avoidance solution.
KEYWORDS
Deadlock avoidance; Arbitrary network topologies/routing policies;
Virtual channels; Regular random graphs
ACM Reference Format:
Jose A. Pascual and Javier Navaridas. 2018. High-Performance, Low-Complexity
Deadlock Avoidance for Arbitrary Topologies/Routings. In Proceedings of
ACM Intl. Conf. on Supercomputing (ICS). ACM, New York, NY, USA, 10 pages.
https://doi.org/10.1145/nnnnnnn.nnnnnnn
1 INTRODUCTION
Exascale computing is the next challenge for the supercomput-
ing community aiming to design systems capable of delivering
Exaops. In order to achieve such a huge computing capability, sys-
tems will require millions of interconnected computing elements
(CE) to execute massive parallel applications. For this reason new
architectures and platforms are being developed, such as our novel,
custom-made architecture ABCD [
11
]. The whole system is com-
posed of tens of millions of low-power-consumption ARM cores
to reach Exascale. These nodes are arranged by means of a uni-
ed, low-latency, lossless interconnection Network (IN) and a fully
distributed storage subsystem with data spread across the nodes.
In such system, the IN is crucial to ensure system performance,
ICS, 2018, Beijing
2018. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM.. . $15.00
https://doi.org/10.1145/nnnnnnn.nnnnnnn
mainly because it needs to scale to extreme levels of parallelism
with applications using tens of thousands of endpoints with any
latency or bandwidth bottlenecks translating into severe penalties
to execution time. In order to meet the requirements of such in-
terconnect in ABCD we are developing our own general purpose
FPGA-based router[
12
]. One of the requirements of our design is
to be simple enough to guarantee low latency while not restrict-
ing the variety of network topologies and routing algorithms we
are currently exploring. This quest for exibility imposes on us
the challenge of developing low complexity deadlock-avoidance
mechanisms able to work with any topology/routing combination.
Such mechanisms traditionally lack of generality being specically
designed for a given topology/routing combo, tightly coupled to the
routing generation process or are based on algorithms whose com-
plexity precludes them for being used in Exascale-sized networks
with millions of endpoints.
In this work we present a collection of three topology- and
routing-agnostic deadlock-avoidance mechanism called Dynamic
Assignment of Virtual Channels (DAVC). DAVC imposes a negli-
gible overhead in terms of logic as it only needs a few registers to
hold local state plus, at most, two comparisons to decide upon tran-
sitions between virtual channels (VCs). In contrast to traditional
topology-agnostic deadlock-avoidance proposals which require pre-
calculation and assignment of paths to VCs, DAVC works on-the-y,
making decisions on each router along the path. Our approach is
completely independent from the topology/routing employed and
does not require to re-calculate and re-assign VCs upon changes
on the architecture, including network failures. The latter is impor-
tant since Exascale systems are expected to have very low mean
time between failures given the sheer number of elements. Hence,
extremely-complex recalculations every relatively small period of
time may render the IN close to useless. DAVC seamlessly works
with arbitrary routing schemes, including minimal, non-minimal
and multipath routing algorithms regardless of them being algorith-
mic, source-routed or table-based. In addition we demonstrate here
that the required number of VCs is bounded by the length of the
longest path. Given that current technology allows for large-radix,
low-diameter topologies – and that the community is following
this very same trend [
4
,
17
,
30
] – the overheads of our proposal
should be relatively small.
First, we prove theoretically that DAVC guarantees deadlock-
freedom for any topology/routing combination. We start by for-
mally dening the deadlock routing problem. Then we show that
the channel allocation induced by our strategies follows a strict
order and thus, it induces and acyclic utilization of channels, which
ensures deadlock freedom. Afterwards, we proceed to evaluate the
1
ICS, 2018, Beijing Pascual and Navaridas
performance of our approach. Given that our focus is on large-
scale interconnects, we rely on simulation to carry out our analy-
sis. We start by assessing DAVC performance against an existing
high-performance network-specic algorithm for the Dragony
topology [
17
]. This mechanism has versions for both Minimal and
Valiant [
33
] routings. Given DAVC exibility, we are able to im-
plement other generic routings which are not supported by the
standard algorithm – in particular, Shortest Path (SP), Equal Cost
Multiple Paths (ECMP) and AllPath (AP). Our second set of exper-
iments uses the Jellysh topology [
30
]–a regular random graph
(RRG)–for which no ecient deadlock-avoidance mechanism exists
[
4
]. For this reason, and given that typical solutions for irregular
networks rely on spanning trees [
24
,
34
] we compare DAVC with a
multi-spanning-tree solution similar to the one proposed in [
22
]
in which a congurable number of spanning trees (one per VC)
are selected. Results show that DAVC delivers similar performance
as the standard deadlock-avoidance mechanism for Dragonies
while allowing the use of other generic routing policies. For RRGs,
DAVC avoids deadlock but delivers much higher performance than
spanning-tree-based solutions.
In summary, the contributions of our paper are the following:
•
We propose a novel, exible, high-performance, low-overhead
deadlock-avoidance mechanisms capable of supporting arbi-
trary network topologies and routing functions.
•
We demonstrate formally that our approach guarantees dead-
lock freedom.
•
We discuss implementation details and highlight the simplic-
ity of its design.
•
We evaluate DAVC against a HPC implementation for Drag-
ony topologies and nd out that it can provide comparable
performance levels than topology-specic approaches.
•
We extend this evaluation to irregular topologies (Jellysh)
where we compare it with a topology-agnostic spanning-tree
based algorithm. DAVC can provide huge benets in terms
of performance and simplicity without all the limitations
and overheads of algorithms that rely on global information.
2 BACKGROUND AND MOTIVATION
Deadlock avoidance has been an active research topic since the very
beginning of HPC INs. There exist basically two types of routing al-
gorithms to create deadlock-free paths: those that avoid the creation
of cycles in the channel dependency graph (CDG) and those that
break the cycles in the CDG using VCs. The most prominent exam-
ples of the rst group are the Spanning Tree protocol, dened in the
IEEE 802.1D Standard [
1
], and Up*/Down* routing [
26
], the standard
in HPC networks such as Inniband. Indeed, Spanning trees are a
specic instance of Up*/Down* routing. Up*/Down* forbids the use
of an up link after a down link has been used. This kind of routing,
mainly used in multi-stage networks (i.e. fat-trees), is deadlock-free
and can be easily implemented without using VCs. However this
approach has many limitations when applied to general topolo-
gies: (i) deciding which links are considered ‘up’ or ‘down’ is far
from trivial, (ii) can leave many resources underutilized (iii) can not
ensure minimal-paths, (iv) routes are not balanced eciently. For
this reason other alternatives such as A-2 and MA-2 routing [
29
],
L-turn routing [
19
] and Multiple Up/Down routing [
10
] have been
proposed to improve the performance of the network by either
increasing the proportion of shortest paths or balancing the use of
the resources. However, they are still essentially simple variations
over the spanning tree concept and so are inherently very restric-
tive in terms of routing and load-balancing. What is worse, they
require some form of topology exploration and embedding global
knowledge into the switching logic, which preclude their use for
large-scale networks. In fact, experimental work around them is
always done with a relatively reduced number of switches (tens of
them, at most).
Regarding the second group of algorithms we can also dier-
entiate between those which decouple the creation of paths from
the deadlock-free assignment to VCs and those which perform
both actions at the same time. DFSSSP [
8
] and LASH [
32
] belong
to the rst group working in a similar way in terms of breaking
cycles searching for them in the CDG and moving individual paths
to other virtual layers. As both techniques can suer from a lim-
ited number of available virtual layers, LASH was improved in
LASH-TOR [
31
] using Up*/Down* routing in the last VC when un-
resolvable cycles appear. Finally, the heuristic approach ACRO [
16
]
was proposed to reduce the number of VCs and the time complex-
ity of both LASH and LASH-TOR. On the other hand BSOR [
18
],
Nue [
7
] and smart routing [
5
] implement a new approach in which
both problems are solved together within the CDG, being able to
impose routing restrictions to the path creation on demand (i.e.
the use of a xed number of VCs). However all of them require to
perform complex searches onto the CDG being the main drawback
of these approaches the computational and memory complexity of
the algorithms.
All the above discussed strategies either lack of generality or
are excessively complex for our purposes, due to the scale we are
aiming at. In addition, none of them is readily available for being
integrated in our environment to compare with DAVC, so, given
their great complexity, we decided not to re-engineer them for our
experimentation purposes.
3 THE DEADLOCK-FREE ROUTING
PROBLEM
In this section we dene the deadlock-free routing problem for
arbitrary network topologies. We start dening terms that will be
used thorough the rest of the paper and giving the conditions that
guarantee a deadlock-free topology/routing combination.
3.1 Denitions
An IN is composed of a set of nodes (computing elements and
switches) with a number of ports. The physical links between nodes
are multiplexed into multiple VCs. These connections are dened
by a
connection rule
which is the function
π
:
N × P → N × P
dened as
π(n, p
n
) = (n
′
, p
n
′
)
which given a node
n ∈ N
and a port
p
n
∈ P
within that node returns the node
n
′
∈ N
and the remote
port p
n
′
∈ P to which is connected to.
Denition 3.1. An IN is a directed graph
I = G(N , C)
in which
N
is the set of nodes and
C
is the set of channels induced by the
connection rule, i.e., given two nodes
n, n
′
∈ N
, the channel
c
n, n
′
∈
C ⇐⇒ ∃p
n
∈ P : π (n, p
n
) = (n
′
, p
n
′
).
2
High-Performance, Low-Complexity Deadlock
Avoidance for Arbitrary Topologies/Routings ICS, 2018, Beijing
0
2
1 3 4
5
0 00 0 0 0
0
1
0 0
1 1
2 2 2
3 3 3
4
5
4 4
5 5
76 8
Figure 1: Example of a network topology showing the iden-
tiers of the computing elements (0–5), the switches (6–8)
and the ports. With colors, we have also represented the se-
quence of ports followed by a packet sent from node 0 to
node 5.
As a consequence, a channel between two nodes
n, n
′
∈ N
is
dened as
c
n, n
′
= ⟨p
n
⟩ = ⟨n, p⟩
such that
π(n, p
n
) = (n
′
, p
n
′
)
. We
dene now a path between two nodes as follows:
Denition 3.2. Given a source node
n
s
and a destination node
n
d
,
a path between
n
s
and
n
d
, dened as
P
n
s
, n
d
= (p
n
0
0
, p
n
1
1
, . . . , p
n
l −1
l−1
) =
(⟨n
0
, p
n
0
0
⟩, ⟨n
1
, p
n
1
1
⟩, . . . , ⟨n
l−1
, p
n
l −1
l−1
⟩)
where
n
i
∈ N
and
p
n
i
i
∈ P
,
is the sequence of ports within each node
n
i
that a packet must
follow to travel from
n
s
= n
0
to
n
d
= n
l−1
. The length of the path,
l, is dened as the number of hops between n
s
and n
d
.
In Fig. 1 we have depicted a path between the nodes 0 and 5
which can be represented as
P
0, 5
= (
0
0
,
4
6
,
5
7
,
3
8
, p
5
c
)
where
p
c
is
the consumption port of the destination node. A generic routing
function
R
assigns the next channel in the path given a destination
node and the current channel:
Denition 3.3. An arbitrary routing function
R
:
N × C → C
for
an IN returns the next channel to be used given the destination node
n
d
and the current channel
c
, i.e.
∀n
d
∈ N , ∃c
′
∈ C
:
R(n
d
, c) = c
′
which is equivalent to R(n
d
, ⟨n, p⟩) = ⟨n
′
, p
′
⟩.
Let us now dene the concepts of inbound port and outbound
port.
Denition 3.4. Given a path
P
and a node
n ∈ N
such that
p
n
i
∈ P
,
we call inbound port of
n
to the port
p
n
′
i−1
and outbound port to the
port p
n
i
.
An example of inbound port and outbound port is depicted in
Fig. 1. If we focus on the third component of the path
p
2
=
5
7
(green
arrow), the inbound port would be
p
1
=
4
6
=
4 and the outbound
port 5. In the same way we dene the concept of outbound node
as the node id to which an outbound port is connected to. In the
previous example the outbound node of the port 5
7
is the node with
id 8.
3.2 Deadlock-free Routing
In this work, we consider a routing function to be valid, if and only
if the paths induced are deadlock-free. Notice that in Denition 3.3,
in opposition to [
7
], we remove the cycle-free and destination-based
conditions from
R
meaning that we are able to deal with cycles in
the paths and with any kind of routing. In [
6
] the authors give the
<0,0>
<1,0>
<2,0>
CGD
n
0
n
1
n
2
p
0
p
0
p
0
CGD
*
<1,0,0> <1,0,1>
<2,0,1><2,0,0>
<0,0,0> <0,0,1>
Figure 2: Simple topology (left) and a representation of all
channel dependencies (middle) and all channel dependen-
cies considering 2 VCs (right).
necessary and sucient condition for a routing to be deadlock-free
(which was reformulated as only a necessary condition in [
28
]).
Next we dene the concept of the channel dependency graph (see
Fig. 2) used by them:
Denition 3.5. A channel dependency graph
D = G(C, E)
is a
directed graph in which the node set
C
is composed by the edge
set of
I
and
E
is the set of edges dened by the routing function
R
such as (c
i
, c
j
) ∈ E ⇐⇒ ∃n ∈ N : R(n, c
i
) = c
j
.
Theorem 3.6. A set of paths within an IN is deadlock-free if and
only if there are no cycles in the corresponding channel dependency
graph.
4 DYNAMIC ASSIGNMENT OF VIRTUAL
CHANNELS
As mentioned before, generic deadlock avoidance strategies try to
break cyces in the CDG. As a result, all of them are applied oine
and then populated into the switches of the IN. The way we tackle
the problem is a completely dierent approach in which cycles are
broken on-the-y while the packets are traversing the network. In
order to dene DAVC we need to redene the concepts of channels
and CDG used in the traditional approaches. We start this section
with some preliminary results which, lately, will be used to proof
that DAVC is deadlock-free for any topology/routing combination.
4.1 Preliminaries
Let us dene the set
S
n
of all tuples
⟨x
1
, x
2
, . . . , x
n
⟩
such that
∀i ∈
{
1
,
2
, . . . , n}
:
x
i
∈ N
and the relation “
<
n
” where
⟨x
1
, x
2
, . . . , x
n
⟩ <
n
⟨y
1
, y
2
, . . . , y
n
⟩ ⇐⇒ ∃i : y
i
> x
i
∧ ∀j ∈ {i + 1, . . . , n} : y
i
= x
i
.
Lemma 4.1. The relation “
<
n
” is a strict order on the set
S
n
of all
tuples ⟨x
1
, x
2
, . . . , x
n
⟩.
Proof.
A relation is a strict order [
27
] if it is irreexive, asym-
metric and transitive. As the demonstration that “
<
n
” fulls those
properties is straightforward we omit the proof. □
Let us consider now an arbitrary graph
D = G(C, E)
where
C
is
the set
S
n
of all tuples of length
n
, and the edge set
E
is induced
by all pairs of nodes
c
i
, c
j
∈ S
n
related through “
<
n
”, such that if
c
i
<
n
c
j
then (e
c
i
, c
j
) ∈ E, that is, D = (S
n
, (S
n
, <
n
)).
Lemma 4.2. D = (S
n
, (S
n
, <
n
)) is a directed acyclic graph.
3
ICS, 2018, Beijing Pascual and Navaridas
Proof.
The proof is straightforward using Lemma 4.1 because
every strict order induces a directed acyclic graph, and hence, the
graph D is acyclic. □
4.2 DAVC Strategy
Let us consider a graph
I = G(N , C
′
)
that represents an arbitrary
topology in which C
′
is the set of channels dened as follows:
Denition 4.3. A channel between two nodes
n, n
′
∈ N
is dened
as
c
n, n
′
= ⟨p
n
, v
p
⟩ = ⟨n, p, v⟩
such that
π(n, p
n
) = (n
′
, p
n
′
)
and
v ∈ V is the virtual channel within ports p
n
and p
n
′
.
Notice that Denition 4.3 extends the denition of channel given
in Section 3.1 to include the VCs. It also implies that there exist
multiple channels between each pair of ports, one per VC. For
example, if the number of VCs is
m
and node
n
is connected to
node
n
′
through port
p
, there exist
m
channels between them:
∀i ∈
{1, 2, . . . , m} : ⟨n, p, v
i
⟩.
Now we dene the function
F
:
N ×C
′
→ C
′
as
F (R(n
d
, ⟨n, p⟩), v) =
⟨n
′
, p
′
, v
′
⟩
and
c <
3
c
′
where
R
is a routing function. Looking at
F
,
paths between nodes have the form
P
n
s
, n
d
= (⟨n
0
, p
n
0
0
, v
p
n
0
0
⟩, . . . , ⟨n
l−1
, p
n
l −1
l−1
, v
p
n
l −1
l −1
l−1
⟩)
which is an increasing strict ordered sequence of channels. We call
F
the allocation function and it is denoted as
F
N P
. The denition
of
F
N P
also implies that the selection of the VCs is independent
from the routing function
R
and that is performed on each router
along the path, after the next hop has been calculated. It is also
easy to view that the CDG induced by
F
N P
is acyclic which implies
that channel transitions generated using
F
N P
are deadlock-free.
We denote this CDG as
CDG
∗
because it uses channels using VCs
(see right part of Fig. 2).
Theorem 4.4. The
CDG
∗
D = (C
′
, E)
in which
C
′
is the set of
channels and
E
is the set of edges induced by the function
F
N P
is
acyclic.
Proof.
It is straightforward to see that
∀c
i
, c
j
∈ C
′
, (c
i
, c
j
) ∈
E ⇐⇒ F
N P
(c
i
) = c
j
=⇒ c
i
<
3
c
j
.
This means that the set
E
is composed of elements of
C
′
which are related through “
<
n
”, so
D = (C
′
, (C
′
, <
3
)), by Lemma 4.2, is acyclic. □
By denition of
<
3
, to order two channels we require the identi-
ers of both current and next channels (node and port ids). However,
instead of using both node and port identiers, we could just use
one of them to select a channel (
⟨n, v
p
n
⟩
or
⟨p
n
, v
p
n
⟩
) and perform
the ordering (VC allocation) using
<
2
. These functions are denoted
as F
N
and F
P
.
Theorem 4.5. The
CDG
∗
D = (C
′
, E)
in which
C
′
is the set of
channels and
E
is the set of edges induced by the functions
F
N
and
F
P
is acyclic.
Proof.
The proof is the same as in Theorem 4.4 but using
<
2
.
□
We conclude this section showing that the allocation functions
are able to deal with loop-paths. Even when these kind of paths are
not desirable, we guarantee that they will not cause deadlocks. This
property will greatly simplify a practical design as will be discussed
later on in Subsection 5.4.
3
4
0
0
2
2 1
7 6
9
1 1 1
F
N
3
4
0
0
2
2 1
7 6
9
0 1 1
F
P
3
4
0
0
2
2 1
7 6
9
0 0 0
F
NP
Figure 3: Examples of the VCs allocation using F
N
(top), F
P
(middle) and F
N P
(bottom) for a given path. Nodes are repre-
sented in blue, ports in green and VCs in red.
Lemma 4.6. A routing function
R
that generates paths which con-
tain loops is deadlock-free if channels are allocated using
F
N P
,
F
N
or
F
P
.
Proof. A path P that contains a loop has the form
P
n
s
, n
d
= (c
0
, . . . , c
i
, . . . , c
j
, . . . , c
i
, . . . , c
l−1
)
in which at least one channel is visited twice, (
c
i
) in the example.
However, we know that by denition of the allocation functions,
c
i
<
2
c
i
or
c
i
<
3
c
i
implies that
⟨n
i
, p
n
i
, v⟩ <
3
⟨n
i
, p
n
i
, v
′
⟩
that is
only possible if
v
′
> v
by denition of
<
3
. This implies that, even
when using the same node and port twice, the path does not create
a loop in the CDG
∗
because the VCs dier. □
In the following sections we analyse DAVC in terms of imple-
mentation and hardware requirements. First, we show how these
allocation functions can be easily implemented on any router with
very low overhead. After this, we perform an analysis of the number
of VCs required to implement them.
5 IMPLEMENTATION OF DAVC
The allocation functions which translate the paths generated by
the routing function into an ordered sequence of channels using
the available VCs can be easily implemented in hardware. As we
will see, the overhead added to the routing process is negligible
requiring a small amount of logic in each router. In Fig. 3 we have
depicted three examples of how channel allocation is performed
using F
N
, F
P
and F
N P
along the same path.
5.1 Node ID based allocation function
We start with the allocation function
F
N
which orders the chan-
nels along a path based only on node identiers. The information
required to perform the VC transition are just the identiers of
the current and the next node in the path. The later is provided
after the routing function has been applied in order to support non-
deterministic routing. The pseudocode to implement this function
is shown in Alg. 1. As we can see Alg. 1 returns the next VC to
be used using the current node identier (currentNID), the current
VC (currentVC) and the outbound node identier (outboundNID)
as dened in Section 3, which is provided by the function getOut-
boundNID(). When the outbound node is lower or equal than the
current one we need to perform a VC transition (+1) to maintain the
order established by
<
2
. In case the outbound identier is higher,
4
