Consistent Hashing Algorithm Based on Slice in Improving Scrapy-Redis Distributed Crawler Efficiency

Author(s):  
Zhe-Yu Ying ◽  
Feng-Li Zhang ◽  
Qing-Yu Fan
2014 ◽  
Vol 602-605 ◽  
pp. 3371-3374
Author(s):  
Peng Wang ◽  
Yan Qi

The NOSQL database to support data and high concurrent read and write,scalability and high availability features in a distributed storage system which has been applied widely. In this paper, through the research of load balancing in distributed storage system,and it proposes the consistent hashing algorithm and the virtual node strategy, in order to improve the load balancing of the system and increase the cache hit ratio. For the load balancing principle of NOSQL and SQL Server, analysis and comparison of the experimental data.The result shows that, with the increase of the number of virtual nodes, the cache hit ratio of NOSQL is higher than the cache hit ratio of SQL Server.


2014 ◽  
Vol 496-500 ◽  
pp. 1812-1816 ◽  
Author(s):  
Hong Xia Mao

In this paper, an extensible system prototype of cloud storage is designed. Considering the load balance of cloud storage system, an improved data storage strategy based on consistent hashing algorithm is proposed in this paper. The strategy adopts virtual nodes to storage data and real–time monitoring of load rate of each storage node to adjust load balancing of the whole storage system. In the improved strategy, the priority is introduced into the storage system to rapidly improve the utilization rate of the new storage nodes. The strategy can effectively optimize the performance of the whole storage system, and improve the overall effect of the load balancing.


Author(s):  
Yihua Lan ◽  
Xiaopu Ma ◽  
Yong Zhang ◽  
Haozheng Ren ◽  
Chao Yin ◽  
...  

2020 ◽  
Vol 2020 ◽  
pp. 1-12
Author(s):  
Jinhua Fu ◽  
Sihai Qiao ◽  
Yongzhong Huang ◽  
Xueming Si ◽  
Bin Li ◽  
...  

Blockchain is widely used in encrypted currency, Internet of Things (IoT), supply chain finance, data sharing, and other fields. However, there are security problems in blockchains to varying degrees. As an important component of blockchain, hash function has relatively low computational efficiency. Therefore, this paper proposes a new scheme to optimize the blockchain hashing algorithm based on PRCA (Proactive Reconfigurable Computing Architecture). In order to improve the calculation performance of hashing function, the paper realizes the pipeline hashing algorithm and optimizes the efficiency of communication facilities and network data transmission by combining blockchains with mimic computers. Meanwhile, to ensure the security of data information, this paper chooses lightweight hashing algorithm to do multiple hashing and transforms the hash algorithm structure as well. The experimental results show that the scheme given in the paper not only improves the security of blockchains but also improves the efficiency of data processing.


Sign in / Sign up

Export Citation Format

Share Document