Can I read Multi-dimensional Parallel Training of Winograd Layer on Memory-Centric Architecture on EtoBox?

Multi-dimensional Parallel Training of Winograd Layer on Memory-Centric Architecture by Byungchul Hong; Yeonju Ro; John Kim is a scholarly article available to read on EtoBox.

What is Multi-dimensional Parallel Training of Winograd Layer on Memory-Centric Architecture about?

Accelerating neural network training is critical in exploring design space of neural networks. Data parallelism is commonly used to accelerate training for Convolutional Neural Networks (CNN) where input batch is distributed across the multiple workers; however, the communication time of weight gradients can limit scalability for moderate batch size. In this work, we propose multi-dimensional parallel training (MPT) of convolution layers by exploiting both data parallelism and intratile parallelism available in Winograd transformed convolution. Workers are organized across two dimensions -one dimension exploiting intra-tile parallelism while the other dimension exploits data parallelism. MPT reduces the amount of communication necessary for weight gradients since weight gradients are only communicated within the data parallelism dimension. However, Winograd transform fundamentally requires more data accesses and the proposed MPT architecture also introduces a new type of communication which we refer to as tile transfergather/scatter of Winograd domain feature maps (tiles). We propose a scalable near-data processing (NDP) architecture to minimize the cost of data accesses through 3D

Author

Byungchul Hong; Yeonju Ro; John Kim

Publisher

IEEE

Published

2018

Language

EN