Deep learning training memory needs can prevent the user from considering large models and large batch sizes. In this work, we propose to use techniques from memory-aware scheduling and automatic differentiation (AD) to execute a backpropagation graph with a bounded memory requirement at the cost of extra recomputations. The case of a single homogeneous chain, i.e. the case of a network whose stages are all identical and form a chain, is well understood and optimal solutions have been proposed in the AD literature. The networks encountered in practice in the context of deep learning are much more diverse, both in terms of shape and heterogeneity. In this work, we define the class of backpropagation graphs, and extend those on which one can compute in polynomial time a solution that minimizes the total number of recomputations. In particular, we consider join graphs which correspond to models such as siamese or cross-modal networks. This article is part of a discussion meeting issue 'Numerical algorithms for high-performance computational science'.

/pdf/optimal-memory-aware-backpropagation-of-deep-join-networks-3q6laf1idq.pdf

Optimal memory-aware backpropagation of deep join networks

A dynamic Tenant-Defined Storage system for efficient resource management in cloud applications

Algorithms for tree-shaped task partition and allocation on heterogeneous multiprocessors

An Elastic Machine (EM) consists of networked fine-grained resources, such as CPU, memory, network and storage, dynamically composed from distributed resource pools offered by a RaaS (Resource-as-a-Service) cloud. EM allocation in RaaS clouds differs from VM placement in IaaS clouds in that it needs to consider the network conditions between the fine-grained resources. To address this problem, this paper presents a d-tree model to represent network conditions for both EM and resource pools and treat EM allocation as a tree packing problem. To solve this NP hard problem efficiently, this paper describes a tree packing framework that combines 12 approximate algorithms by using tree density to sort d-trees and virtual distance to filter d-trees. Using simulation tests and 6 quality measures, new algorithms that outperform the previous tree packing algorithm are discovered. Furthermore, the tests show that the algorithms that respect the network conditions outperform those that ignore them in most cases. Moreover, the top 2 algorithms are identified using a ranking function that combines the quality measures.

Tree Packing for Elastic Machine Allocation in RaaS Cloud

Exact and approximation algorithms for synthesizing specific classes of optimal block-structured processes

https://hal.inria.fr/hal-01397299/document

Scheduling Series-Parallel Task Graphs to Minimize Peak Memory

https://hal.inria.fr/hal-01071414/document

Thomas Lambert

Papers

Scheduling Series-Parallel Task Graphs to Minimize Peak Memory

Comments on the hierarchically structured bin packing problem