scholarly journals A Framework for Distributed Data-Parallel Execution in the Kepler Scientific Workflow System

2012 ◽  
Vol 9 ◽  
pp. 1620-1629 ◽  
Author(s):  
Jianwu Wang ◽  
Daniel Crawl ◽  
Ilkay Altintas
Keyword(s):  
Scientific Workflow ◽  
Parallel Execution ◽  
Distributed Data ◽  
Data Parallel ◽  
Workflow System

Process-oriented ecological modeling approach and scientific workflow system

2014 ◽  
Vol 22 (3) ◽  
pp. 277
Author(s):  
Qiao Huijie ◽  
Lin Congtian ◽  
Wang Jiangning ◽  
Ji Liqiang
Keyword(s):  
Ecological Modeling ◽  
Scientific Workflow ◽  
Modeling Approach ◽  
Workflow System ◽  
Process Oriented

scholarly journals Application Scenarios Using Serpens Suite for Kepler Scientific Workflow System

2012 ◽  
Vol 9 ◽  
pp. 1604-1613 ◽  
Author(s):  
Marcin Płóciennik ◽  
Michał Owsiak ◽  
Tomasz Zok ◽  
Bartek Palak ◽  
Antonio Gómez-Iglesias ◽  
...  
Keyword(s):  
Scientific Workflow ◽  
Workflow System

scholarly journals Early Cloud Experiences with the Kepler Scientific Workflow System

2012 ◽  
Vol 9 ◽  
pp. 1630-1634 ◽  
Author(s):  
Jianwu Wang ◽  
Ilkay Altintas
Keyword(s):  
Scientific Workflow ◽  
Workflow System

scholarly journals On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning

2020 ◽  
Vol 34 (04) ◽  
pp. 3817-3824
Author(s):  
Aritra Dutta ◽  
El Houcine Bergou ◽  
Ahmed M. Abdelmoniem ◽  
Chen-Yu Ho ◽  
Atal Narayan Sahu ◽  
...  
Keyword(s):  
Deep Learning ◽  
Theoretical Analysis ◽  
Deep Neural Networks ◽  
Theory And Practice ◽  
Distributed Data ◽  
Communication Costs ◽  
Data Parallel ◽  
Model Compression ◽  
Wide Range ◽  
Learning Frameworks

Compressed communication, in the form of sparsification or quantization of stochastic gradients, is employed to reduce communication costs in distributed data-parallel training of deep neural networks. However, there exists a discrepancy between theory and practice: while theoretical analysis of most existing compression methods assumes compression is applied to the gradients of the entire model, many practical implementations operate individually on the gradients of each layer of the model.In this paper, we prove that layer-wise compression is, in theory, better, because the convergence rate is upper bounded by that of entire-model compression for a wide range of biased and unbiased compression methods. However, despite the theoretical bound, our experimental study of six well-known methods shows that convergence, in practice, may or may not be better, depending on the actual trained model and compression ratio. Our findings suggest that it would be advantageous for deep learning frameworks to include support for both layer-wise and entire-model compression.

Sign in / Sign up

Export Citation Format

Share Document