End-to-End Dataflow Parallelism for Transfer Throughput Optimization

2011 ◽  
pp. 23-39
Author(s):  
Esma Yildirim ◽  
Tevfik Kosar
Keyword(s):  
High Speed ◽  
Large Scale ◽  
Data Transfer ◽  
Throughput Optimization ◽  
High Speed Networks ◽  
Parallel Storage ◽  
End To End ◽  
Processor Speed ◽  
Networking Technology

The emerging petascale increase in the data produced by large-scale scientific applications necessitates innovative solutions for efficient transfer of data through the advanced infrastructure provided by today’s high-speed networks and complex computer-architectures (e.g. supercomputers, parallel storage systems). Although the current optical networking technology reached transport speeds of 100Gbps, the applications still suffer from the inadequate transport protocols and end-system bottlenecks such as processor speed, disk I/O speed and network interface card limits that cause underutilization of the existing network infrastructure and let the application achieve only a small portion of the theoretical performance. Fortunately, with the parallelism provided by usage of multiple CPUs/nodes and multiple disks present in today’s systems, these bottlenecks could be eliminated. However it is necessary to understand the characteristics of the end-systems and the transport protocol used. In this book chapter, we analyze methodologies that will improve the data transfer speed of applications and provide maximal speeds that could be obtained from the available end-system resources and high-speed networks through usage of end-to-end dataflow parallelism.

OPTIMIZED METHOD OF NETWORK PACKET ROUTING

2019 ◽  
pp. 27-32
Author(s):  
A. S. Kotlyarov
Keyword(s):  
High Speed ◽  
Large Scale ◽  
Data Transfer ◽  
Search Method ◽  
Concurrent Use ◽  
Parallel Pipeline ◽  
Field Programmable ◽  
Modified Block ◽  
Hardware Costs ◽  
Routing Tables

In the paper we review the possibility of using a block search method for network packets routing tasks in high-speed computer networks. The method provides minimization of hardware costs for large-scale routing tables. To support the maximum data transfer rate, it is necessary to perform real-time routing of packets. The existing solution presumes concurrent use of several routing devices. Each device performs independent search of records by the bisection method. However, if the channel rate exceeds 10 Gb/sec, and the number of routs exceeds 220, it leads to high hardware costs. To reduce hardware costs, we have suggested to use a modified block search method, which differs from the classic one by parallel-pipeline form of search. We have presented evaluation of the minimum field-programmable gate array hardware costs for the network packets routing task. Analysis of the results has proved efficiency of the suggested method in comparison with existing solutions. As a result, the hardware costs were reduced in 5 times.

scholarly journals Cloud bursting galaxy: federated identity and access management

2019 ◽  
Vol 36 (1) ◽  
pp. 1-9 ◽  
Author(s):  
Vahid Jalili ◽  
Enis Afgan ◽  
James Taylor ◽  
Jeremy Goecks
Keyword(s):  
Cloud Computing ◽  
High Speed ◽  
Large Scale ◽  
Best Practice ◽  
Data Transfer ◽  
Data Access ◽  
Web Security ◽  
Biomedical Data ◽  
Authentication And Authorization ◽  
Computing Platforms

Abstract Motivation Large biomedical datasets, such as those from genomics and imaging, are increasingly being stored on commercial and institutional cloud computing platforms. This is because cloud-scale computing resources, from robust backup to high-speed data transfer to scalable compute and storage, are needed to make these large datasets usable. However, one challenge for large-scale biomedical data on the cloud is providing secure access, especially when datasets are distributed across platforms. While there are open Web protocols for secure authentication and authorization, these protocols are not in wide use in bioinformatics and are difficult to use for even technologically sophisticated users. Results We have developed a generic and extensible approach for securely accessing biomedical datasets distributed across cloud computing platforms. Our approach combines OpenID Connect and OAuth2, best-practice Web protocols for authentication and authorization, together with Galaxy (https://galaxyproject.org), a web-based computational workbench used by thousands of scientists across the world. With our enhanced version of Galaxy, users can access and analyze data distributed across multiple cloud computing providers without any special knowledge of access/authorization protocols. Our approach does not require users to share permanent credentials (e.g. username, password, API key), instead relying on automatically generated temporary tokens that refresh as needed. Our approach is generalizable to most identity providers and cloud computing platforms. To the best of our knowledge, Galaxy is the only computational workbench where users can access biomedical datasets across multiple cloud computing platforms using best-practice Web security approaches and thereby minimize risks of unauthorized data access and credential use. Availability and implementation Freely available for academic and commercial use under the open-source Academic Free License (https://opensource.org/licenses/AFL-3.0) from the following Github repositories: https://github.com/galaxyproject/galaxy and https://github.com/galaxyproject/cloudauthz.

Sign in / Sign up

Export Citation Format

Share Document