Turbine: A Distributed-memory Dataflow Engine for High Performance Many-task Applications.

Saved in:

Bibliographic Details
Title:	Turbine: A Distributed-memory Dataflow Engine for High Performance Many-task Applications.
Authors:	Wozniak, Justin M.¹ wozniak@mcs.anl.gov, Armstrong, Timothy G.² tga@uchicago.edu, Maheshwari, Ketan¹ ketan@mcs.anl.gov, Lusk, Ewing L.¹ lusk@mcs.anl.gov, Katz, Daniel S.³ d.katz@ieee.org, Wilde, Michael¹ wilde@mcs.anl.gov, Foster, Ian T.¹ foster@mcs.anl.gov
Source:	Fundamenta Informaticae. 2013, Vol. 128 Issue 3, p337-366. 30p.
Subjects:	Turbines, Distributed shared memory, High performance computing, Application software, Computer programming, Computer performance, Data flow computing
Abstract:	Efficiently utilizing the rapidly increasing concurrency of multi-petaflop computing systems is a significant programming challenge. One approach is to structure applications with an upper layer of many loosely coupled coarse-grained tasks, each comprising a tightly-coupled parallel function or program. 'Many-task' programming models such as functional parallel dataflow may be used at the upper layer to generate massive numbers of tasks, each of which generates significant tightly coupled parallelism at the lower level through multithreading, message passing, and/or partitioned global address spaces. At large scales, however, the management of task distribution, data dependencies, and intertask data movement is a significant performance challenge. In this work, we describe Turbine, a new highly scalable and distributed many-task dataflow engine. Turbine executes a generalized many-task intermediate representation with automated self-distribution and is scalable to multi-petaflop infrastructures. We present here the architecture of Turbine and its performance on highly concurrent systems. [ABSTRACT FROM AUTHOR]
	Copyright of Fundamenta Informaticae is the property of Polskie Towarzystwo Matematyczne and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database:	Engineering Source

Description
Abstract:	Efficiently utilizing the rapidly increasing concurrency of multi-petaflop computing systems is a significant programming challenge. One approach is to structure applications with an upper layer of many loosely coupled coarse-grained tasks, each comprising a tightly-coupled parallel function or program. 'Many-task' programming models such as functional parallel dataflow may be used at the upper layer to generate massive numbers of tasks, each of which generates significant tightly coupled parallelism at the lower level through multithreading, message passing, and/or partitioned global address spaces. At large scales, however, the management of task distribution, data dependencies, and intertask data movement is a significant performance challenge. In this work, we describe Turbine, a new highly scalable and distributed many-task dataflow engine. Turbine executes a generalized many-task intermediate representation with automated self-distribution and is scalable to multi-petaflop infrastructures. We present here the architecture of Turbine and its performance on highly concurrent systems. [ABSTRACT FROM AUTHOR]
ISSN:	01692968
DOI:	10.3233/FI-2013-949