Data Chunks Placement Optimization for Hybrid Storage Systems

“Hybrid Cloud Storage” (HCS) is a widely adopted framework that combines the functionality of public and private cloud storage models to provide storage services. This kind of storage is especially ideal for organizations that seek to reduce the cost of their storage infrastructure with the use of “Public Cloud Storage” as a backend to on-premises primary storage. Despite the higher performance, the hybrid cloud has latency issues, related to the distance and bandwidth of the public storage, which may cause a significant drop in the performance of the storage systems during data transfer. This issue can become a major problem when one or more private storage nodes fail. In this paper, we propose a new framework for optimizing the data uploading process that is currently used with hybrid cloud storage systems. The optimization is concerned with spreading the data over the multiple storages in the HCS system according to some predefined objective functions. Furthermore, we also used Network Coding technics for minimizing data transfer latency between the receiver (private storages) and transmitter nodes.

An Authorized Clou Dedup in Hybrid Cloud using Triple Data Encryption Standard

Nowadays, data deduplication has become more essential for cloud storage providers because of continuous increase in number of users and their data file size. The users are allowed to access server anytime and anywhere to upload/download their data file. Whenever data is retrieving, it leads to several problems associated to the confidentiality and privacy. For protection of data security, we proposed an efficient technique called ClouDedup which assures file deduplication. To secure the confidentiality of critical data while supporting ClouDedup checker, we proposed a triple data encryption standard (TDES) technique to encrypt the data prior to uploading the data file in cloud storage. The privilege level of user is verified with data to assure whether he is an authorized user or not. The analysis of security demonstrates that our proposed security method is safe and secure. We prove that our proposed ClouDedup method has minimal overhead compared to normal operations. The process aims to use authorized ClouDedup checker with a triple data encryption standard (TDES) technique to minimize duplication copies of data in hybrid cloud storage and conducted test experiments using our prototype.

Hybrid Cloud Storage for Secure Authorization and Information Hiding

Distributed database plays a vital feature in every day life because of the fact inside the present technology, commercial organization environment is growing at very fast fee so our fundamental desire is to get reliable statistics from any supply. Since our database is sent, way facts is placed at exceptional geographical locations and sooner or later lets in to without issues access our precious & precious information. We advise an architecture that integrates cloud database offerings with information integrity and the opportunity of executing concurrent operations on encrypted statistics. It is the solution helping geographically dispensed customers to attach without delay to an encrypted cloud database, and to execute the concurrent and the impartial operations together with the ones editing the database shape. Distributed database is the emerging technique which focuses on concurrency control and safety problems underneath this allocated database. In this studies work, information safety is greater via the usage of NTRU (N-th degree Truncated polynomial Ring Unit or Number Theory Research Unit) uneven key set of rules in which the wonderful keys are used for encryption of plaintext and decryption of cipher text. These keys are named as public and private keys. NTRU being speedy and cozy hashing set of rules to be able to offer more security to the gadget, in terms of throughput and their processing tempo. Its essential traits are the low memory and computational necessities as providing a immoderate security degree. It is a totally well-prepared public-key cryptosystem. MD5 hash feature is also used for checking data integrity sooner or later of the authentication manner

Preprocessing Big Data for Efficient Storage and Research

Big Data refers to large datasets and so it is not possible to store, manage and analyze it using commonly used software systems. The emergence of smart phones, social networks and online applications has led to the generation of massive amounts of structured, unstructured and semi structured data. Big data analytics has received sizeable attention since it offers a great opportunity to uncover potentials from heavy amounts of data. Data preprocessing techniques, when applied prior to analytics, can substantially improve the overall quality of the patterns mined and/or the time required for the actual mining. Thus this paper presents an efficient method for preprocessing data and also partitioning big dataset based on sensitivity parameters. The partitioned dataset can be uploaded to public and private cloud based on the importance of data in the partition. Thus hybrid cloud storage and processing of big data is supported by this approach. The experimental results show that the proposed method preprocesses and partition data with high accuracy and reduced processing time.

COSA: Cloud Object Storage Archive for deep archival of digital data

The Cornell Lab of Ornithology gathers, utilizes and archives a wide variety of digital assets ranging from details of a bird observation to photos, video and sound recordings. Some of these datasets are fairly small, while others are hundreds of terabytes. In this presentation we will describe how the Lab archives these datasets to ensure the data are both loss-less and recoverable in the case of a widespread disaster, how the archival strategy has evolved over the years and explore in detail the current hybrid cloud storage management system. The Lab runs eBird and several other citizen science programs focused on birds where individuals from around the globe enter their sightings into a centralized database. The eBird project alone stores over 500,000,000 observations and the underlying database is over a terabyte in size. Birds of North America, Neotropical Birds and All About Birds are online species accounts comprising a wide range of authoritative live history articles maintained in a relatively small database. Macaulay Library is the world’s largest image, sound and video archive with over 6,000,000 cuts totaling nearly 100 TB of data. The Bioacoustics Research Program utilizes automated recording units (SWIFTs) in the forests of the US, jungles of Africa and in all seven oceans to record the environment. These units record 24 hours a day and gather a tremendous about of raw data, over 200 TB to date with an expected rate of an additional 100TB per year. Lastly, BirdCams run by the lab add a steady stream of media detailing the reproductive cycles of a number of species. The lab is committed to making these archives of the natural world available for research and conservation today. More importantly, ensuring these data exist and are accessible in 100 years is a critical component of the Lab data strategy. The data management system for these digital assets has been completely overhauled to handle the rapidly increasing volume and to utilize on-premises systems and cloud services in a hybrid cloud storage system to ensure data are archived in a manner that is redundant, loss-less and insulated from disasters yet still accessible for research. With multimedia being the largest and most rapidly growing block of data, cost rapidly becomes a constraining factor of archiving these data in redundant, geographically isolated facilities. Datasets with a smaller footprint, eBIrd and species accounts allow for a wider variety of solutions as cost is less of a factor. Using different methods to take advantage of differing technologies and balancing cost vs recovery speed, the Lab has implemented several strategies based on data stability (eBird data are constantly changing), retrieval frequency required for research and overall size of the dataset. We utilize Amazon S3 and Glacier as our media archive, we tag each media in Glacier with a set of basic DarwinCore metatdata fields that key back to a master metadata database and numerous project specific databases. Because these metadata databases are much smaller in size, yet critical in searching and retrieval of a required media file, they are archived differently with up to the minute replication to prevent any data loss due to an unexpected disaster. The media files are tagged with a standard set of basic metadata and in the case where the metadata databases were unavailable, retrieval of specific media and basic metadata can still occur. This system has allowed the lab to place into long term archive hundreds of terabytes of data, store them in redundant, geographically isolated locations and provide for complete disaster recovery of the data and metadata.