Load Balancing Using Consistent Hashing: A Real Challenge for Large Scale Distributed Web Crawlers

Abstract We present PeerFlow, a system to securely load balance client traffic in Tor. Security in Tor requires that no adversary handle too much traffic. However, Tor relays are run by volunteers who cannot be trusted to report the relay bandwidths, which Tor clients use for load balancing. We show that existing methods to determine the bandwidths of Tor relays allow an adversary with little bandwidth to attack large amounts of client traffic. These methods include Tor’s current bandwidth-scanning system, TorFlow, and the peer-measurement system EigenSpeed. We present an improved design called PeerFlow that uses a peer-measurement process both to limit an adversary’s ability to increase his measured bandwidth and to improve accuracy. We show our system to be secure, fast, and efficient. We implement PeerFlow in Tor and demonstrate its speed and accuracy in large-scale network simulations.

Download Full-text

Periodic hierarchical load balancing for large supercomputers

The International Journal of High Performance Computing Applications ◽

10.1177/1094342010394383 ◽

2011 ◽

Vol 25 (4) ◽

pp. 371-385 ◽

Cited By ~ 34

Author(s):

Gengbin Zheng ◽

Abhinav Bhatelé ◽

Esteban Meneses ◽

Laxmikant V. Kalé

Keyword(s):

Load Balancing ◽

Large Scale ◽

Parallel Machines ◽

National Laboratory ◽

Argonne National Laboratory ◽

Parallel Applications ◽

Scientific Application ◽

Computing Center ◽

Blue Gene ◽

Advanced Computing

Large parallel machines with hundreds of thousands of processors are becoming more prevalent. Ensuring good load balance is critical for scaling certain classes of parallel applications on even thousands of processors. Centralized load balancing algorithms suffer from scalability problems, especially on machines with a relatively small amount of memory. Fully distributed load balancing algorithms, on the other hand, tend to take longer to arrive at good solutions. In this paper, we present an automatic dynamic hierarchical load balancing method that overcomes the scalability challenges of centralized schemes and longer running times of traditional distributed schemes. Our solution overcomes these issues by creating multiple levels of load balancing domains which form a tree. This hierarchical method is demonstrated within a measurement-based load balancing framework in Charm++. We discuss techniques to deal with scalability challenges of load balancing at very large scale. We present performance data of the hierarchical load balancing method on up to 16,384 cores of Ranger (at the Texas Advanced Computing Center) and 65,536 cores of Intrepid (the Blue Gene/P at Argonne National Laboratory) for a synthetic benchmark. We also demonstrate the successful deployment of the method in a scientific application, NAMD, with results on Intrepid.

Download Full-text

E-Learning at Politecnico di Torino

Handbook of Research on Didactic Strategies and Technologies for Education ◽

10.4018/978-1-4666-2122-0.ch060 ◽

2013 ◽

pp. 690-702

Author(s):

Salvatore Barbagallo ◽

Roberto Bertonasco ◽

Fulvio Corno ◽

Laura Farinetti ◽

Marco Mezzalama ◽

...

Keyword(s):

Distance Education ◽

Large Scale ◽

Sustainable Use ◽

First Year ◽

Preliminary Evaluation ◽

Full Time ◽

Large Scale System ◽

Part Time ◽

E Learning ◽

Real Challenge

Politecnico di Torino has been actively experimenting distance education scenarios since 1992, through the development of innovative methodologies and tools. The real challenge today, however, is to move from small settings to a large-scale system able to suit the needs of a broad number of users belonging to different categories, from traditional students to part-time or full-time workers, from students living far from Torino to people with participation restriction due to disability. The emphasis then, is not only on the innovation of methodologies and technologies, but on their effective and economically sustainable use in a complex and multi-faceted setting. This chapter describes the services introduced in this direction and gives a preliminary evaluation after the first year of delivery.

Download Full-text

Load Balancing to Increase the Consistency of Replicas in Data Grids

Technology Integration Advancements in Distributed Systems and Computing ◽

10.4018/978-1-4666-0906-8.ch003 ◽

2012 ◽

pp. 42-57

Author(s):

Ghalem Belalem ◽

Naima Belayachi ◽

Radjaa Behidji ◽

Belabbes Yagoubi

Keyword(s):

Load Balancing ◽

Large Scale ◽

Data Grids ◽

Distributed Resources ◽

Data Set ◽

Large Scale Systems ◽

Consistency Management ◽

Geographically Distributed ◽

Replication Technique ◽

Improve Service Quality

Data grids are current solutions to the needs of large scale systems and provide a set of different geographically distributed resources. Their goal is to offer an important capacity of parallel calculation, ensure a data effective and rapid access, improve the availability, and tolerate the breakdowns. In such systems, however, these advantages are possible only by using the replication technique. The use of this technique raises the problem of maintaining consistency of replicas of the same data set. In order to guarantee replica set reliability, it is necessary to have high coherence. This fact, however, penalizes performance. In this paper, the authors propose studying balancing influence on replica quality. For this reason, a service of hybrid consistency management is developed, which combines the pessimistic and optimistic approaches and is extended by a load balancing service to improve service quality. This service is articulated on a hierarchical model with two levels.

Download Full-text

Peer-to-Peer Service Discovery for Grid Computing

Grid and Cloud Computing ◽

10.4018/978-1-4666-0879-5.ch111 ◽

2012 ◽

pp. 232-259

Author(s):

Eddy Caron ◽

Frédéric Desprez ◽

Franck Petit ◽

Cédric Tedeschi

Keyword(s):

Fault Tolerance ◽

Load Balancing ◽

Service Discovery ◽

Large Scale ◽

Dynamic Networks ◽

Peer To Peer ◽

Indexing System ◽

Key Points ◽

Computing Platforms ◽

Traditional Approaches

Within distributed computing platforms, some computing abilities (or services) are offered to clients. To build dynamic applications using such services as basic blocks, a critical prerequisite is to discover those services. Traditional approaches to the service discovery problem have historically relied upon centralized solutions, unable to scale well in large unreliable platforms. In this chapter, we will first give an overview of the state of the art of service discovery solutions based on peer-to-peer (P2P) technologies that allow such a functionality to remain efficient at large scale. We then focus on one of these approaches: the Distributed Lexicographic Placement Table (DLPT) architecture, that provide particular mechanisms for load balancing and fault-tolerance. This solution centers around three key points. First, it calls upon an indexing system structured as a prefix tree, allowing multi-attribute range queries. Second, it allows the mapping of such structures onto heterogeneous and dynamic networks and proposes some load balancing heuristics for it. Third, as our target platform is dynamic and unreliable, we describe its powerful fault-tolerance mechanisms, based on self-stabilization. Finally, we present the software prototype of this architecture and its early experiments.

Download Full-text

Scheduling issue for Dynamic Load Balancing of map-reduce in large scale data (Big data)

Journal of Xidian University ◽

10.37896/jxu14.5/445 ◽

2020 ◽

Vol 14 (5) ◽

Keyword(s):

Big Data ◽

Load Balancing ◽

Dynamic Load ◽

Large Scale ◽

Dynamic Load Balancing ◽

Map Reduce ◽

Large Scale Data ◽

Scale Data

Download Full-text

Optimal User Association Strategy for Large-Scale IoT Sensor Networks with Mobility on Cloud RANs

Sensors ◽

10.3390/s19204415 ◽

2019 ◽

Vol 19 (20) ◽

pp. 4415 ◽

Cited By ~ 1

Author(s):

Taewoon Kim ◽

Chanjun Chun ◽

Wooyeol Choi

Keyword(s):

Load Balancing ◽

Large Scale ◽

Mobile Internet ◽

Base Stations ◽

Traffic Load ◽

User Association ◽

Network Simulator ◽

Traffic Loads ◽

Association Strategy ◽

Radio Access

In networking systems such as cloud radio access networks (C-RAN) where users receive the connection and data service from short-range, light-weight base stations (BSs), users’ mobility has a significant impact on their association with BSs. Although communicating with the closest BS may yield the most desirable channel conditions, such strategy can lead to certain BSs being over-populated while leaving remaining BSs under-utilized. In addition, mobile users may encounter frequent handovers, which imposes a non-negligible burden on BSs and users. To reduce the handover overhead while balancing the traffic loads between BSs, we propose an optimal user association strategy for a large-scale mobile Internet of Things (IoT) network operating on C-RAN. We begin with formulating an optimal user association scheme focusing only on the task of load balancing. Thereafter, we revise the formulation such that the number of handovers is minimized while keeping BSs well-balanced in terms of the traffic load. To evaluate the performance of the proposed scheme, we implement a discrete-time network simulator. The evaluation results show that the proposed optimal user association strategy can significantly reduce the number of handovers, while outperforming conventional association schemes in terms of load balancing.

Download Full-text