scholarly journals Dataset-adaptive minimizer order reduces memory usage in k-mer counting

2021 ◽  
Author(s):  
Dan Flomin ◽  
David Pellow ◽  
Ron Shamir
Keyword(s):  
Deep Sequencing ◽  
Load Distribution ◽  
State Of The Art ◽  
Memory Usage ◽  
Continuous Growth ◽  
Memory Footprint ◽  
Specific Order ◽  
Order Change ◽  
Memory Reduction ◽  
Memory Efficient

AbstractThe rapid, continuous growth of deep sequencing experiments requires development and improvement of many bioinformatics applications for analysis of large sequencing datasets, including k-mer counting and assembly. Several applications reduce RAM usage by binning sequences. Binning is done by employing minimizer schemes, which rely on a specific order of the minimizers. It has been demonstrated that the choice of the order has a major impact on the performance of the applications. Here we introduce a method for tailoring the order to the dataset. Our method repeatedly samples the dataset and modifies the order so as to flatten the k-mer load distribution across minimizers. We integrated our method into Gerbil, a state-of-the-art memory efficient k-mer counter, and were able to reduce its memory footprint by 50% or more for large k, with only minor increase in runtime. Our tests also showed that the orders produced by our method produced superior results when transferred across datasets from the same species, with little or no order change. This enables memory reduction with essentially no increase in runtime.

scholarly journals Robust CNN Compression Framework for Security-Sensitive Embedded Systems

2021 ◽  
Vol 11 (3) ◽  
pp. 1093
Author(s):  
Jeonghyun Lee ◽  
Sangkyun Lee
Keyword(s):  
Embedded Systems ◽  
Optimization Problem ◽  
State Of The Art ◽  
Classification Problems ◽  
Proximal Gradient Method ◽  
Knowledge Distillation ◽  
New Type ◽  
Adversarial Examples ◽  
Adversarial Training ◽  
Memory Efficient

Convolutional neural networks (CNNs) have achieved tremendous success in solving complex classification problems. Motivated by this success, there have been proposed various compression methods for downsizing the CNNs to deploy them on resource-constrained embedded systems. However, a new type of vulnerability of compressed CNNs known as the adversarial examples has been discovered recently, which is critical for security-sensitive systems because the adversarial examples can cause malfunction of CNNs and can be crafted easily in many cases. In this paper, we proposed a compression framework to produce compressed CNNs robust against such adversarial examples. To achieve the goal, our framework uses both pruning and knowledge distillation with adversarial training. We formulate our framework as an optimization problem and provide a solution algorithm based on the proximal gradient method, which is more memory-efficient than the popular ADMM-based compression approaches. In experiments, we show that our framework can improve the trade-off between adversarial robustness and compression rate compared to the existing state-of-the-art adversarial pruning approach.

scholarly journals Fast, Small and Exact: Infinite-order Language Modelling with Compressed Suffix Trees

2016 ◽  
Vol 4 ◽  
pp. 477-490 ◽  
Author(s):  
Ehsan Shareghi ◽  
Matthias Petri ◽  
Gholamreza Haffari ◽  
Trevor Cohn
Keyword(s):  
Infinite Order ◽  
State Of The Art ◽  
Language Model ◽  
Language Models ◽  
Memory Usage ◽  
Suffix Trees ◽  
Construction Time ◽  
Modest Increase ◽  
Order Language ◽  
Language Modelling

Efficient methods for storing and querying are critical for scaling high-order m-gram language models to large corpora. We propose a language model based on compressed suffix trees, a representation that is highly compact and can be easily held in memory, while supporting queries needed in computing language model probabilities on-the-fly. We present several optimisations which improve query runtimes up to 2500×, despite only incurring a modest increase in construction time and memory usage. For large corpora and high Markov orders, our method is highly competitive with the state-of-the-art KenLM package. It imposes much lower memory requirements, often by orders of magnitude, and has runtimes that are either similar (for training) or comparable (for querying).

Efficiency Optimized Pneumatic Pressure Booster

2019 ◽  
Author(s):  
Olivier Reinertz ◽  
Katharina Schmitz
Keyword(s):  
Load Distribution ◽  
State Of The Art ◽  
Analytical Description ◽  
The Novel ◽  
One Dimensional ◽  
Pneumatic Pressure ◽  
Novel Device ◽  
Working Cycle ◽  
Promising Solution ◽  
Dimensional Simulation

Abstract In the scope of this paper, a novel efficiency optimized supply pressure adaptive concept of pneumatic pressure boosters is presented. It is deduced from a profound analysis of state of the art components. The working cycle of the pump chambers can be divided into a filling, compression, pumping and decompression phase. A promising solution for efficiency improvements, which is further analyzed in the scope of this paper, is to adapt the required force of the compression chambers by nonlinear mechanics. Thus, a smaller force at the end of the stroke is required and a reduced air consumption of the driving chamber occurs. As the force demand of the compression chamber and therewith the load distribution over the stroke changes with the operational pressures, an adaptive concept needs to be implemented. The novel device and its parameterization are deduced by means of an analytical description of state of the art pressure boosters. Subsequently, it is investigated by one-dimensional simulation in DSHplus. The results show broad applicability of the method in relevant applications and huge energy saving potentials compared to state of the art products.

iHDT++: improving HDT for SPARQL triple pattern resolution

2020 ◽  
Vol 39 (2) ◽  
pp. 2249-2261
Author(s):  
Antonio Hernández-Illera ◽  
Miguel A. Martínez-Prieto ◽  
Javier D. Fernández ◽  
Antonio Fariña
Keyword(s):  
State Of The Art ◽  
Simple Design ◽  
Performance Improvements ◽  
Semantic Data ◽  
Object Based ◽  
Efficient Access ◽  
Memory Footprint ◽  
Significant Performance ◽  
Pattern Resolution

RDF self-indexes compress the RDF collection and provide efficient access to the data without a previous decompression (via the so-called SPARQL triple patterns). HDT is one of the reference solutions in this scenario, with several applications to lower the barrier of both publication and consumption of Big Semantic Data. However, the simple design of HDT takes a compromise position between compression effectiveness and retrieval speed. In particular, it supports scan and subject-based queries, but it requires additional indexes to resolve predicate and object-based SPARQL triple patterns. A recent variant, HDT++, improves HDT compression ratios, but it does not retain the original HDT retrieval capabilities. In this article, we extend HDT++ with additional indexes to support full SPARQL triple pattern resolution with a lower memory footprint than the original indexed HDT (called HDT-FoQ). Our evaluation shows that the resultant structure, iHDT++ , requires 70 - 85% of the original HDT-FoQ space (and up to 48 - 72% for an HDT Community variant). In addition, iHDT++ shows significant performance improvements (up to one level of magnitude) for most triple pattern queries, being competitive with state-of-the-art RDF self-indexes.

scholarly journals A survey on compact features for visual content analysis

2016 ◽  
Vol 5 ◽  
Author(s):  
Luca Baroffio ◽  
Alessandro E. C. Redondi ◽  
Marco Tagliasacchi ◽  
Stefano Tubaro
Keyword(s):  
Content Analysis ◽  
Large Scale ◽  
Ad Hoc ◽  
State Of The Art ◽  
Large Body ◽  
Visual Features ◽  
Low Latency ◽  
Visual Content ◽  
Feature Encoding ◽  
Memory Footprint

Visual features constitute compact yet effective representations of visual content, and are being exploited in a large number of heterogeneous applications, including augmented reality, image registration, content-based retrieval, and classification. Several visual content analysis applications are distributed over a network and require the transmission of visual data, either in the pixel or in the feature domain, to a central unit that performs the task at hand. Furthermore, large-scale applications need to store a database composed of up to billions of features and perform matching with low latency. In this context, several different implementations of feature extraction algorithms have been proposed over the last few years, with the aim of reducing computational complexity and memory footprint, while maintaining an adequate level of accuracy. Besides extraction, a large body of research addressed the problem of ad-hoc feature encoding methods, and a number of networking and transmission protocols enabling distributed visual content analysis have been proposed. In this survey, we present an overview of state-of-the-art methods for the extraction, encoding, and transmission of compact features for visual content analysis, thoroughly addressing each step of the pipeline and highlighting the peculiarities of the proposed methods.

scholarly journals Beyond equi-joins

2021 ◽  
Vol 14 (11) ◽  
pp. 2599-2612
Author(s):  
Nikolaos Tziavelis ◽  
Wolfgang Gatterbauer ◽  
Mirek Riedewald
Keyword(s):  
Experimental Study ◽  
State Of The Art ◽  
Database Systems ◽  
Ranking Function ◽  
Space Complexity ◽  
Time And Space ◽  
Running Time ◽  
Join Queries ◽  
Time And Space Complexity ◽  
Memory Efficient

We study theta-joins in general and join predicates with conjunctions and disjunctions of inequalities in particular, focusing on ranked enumeration where the answers are returned incrementally in an order dictated by a given ranking function. Our approach achieves strong time and space complexity properties: with n denoting the number of tuples in the database, we guarantee for acyclic full join queries with inequality conditions that for every value of k , the k top-ranked answers are returned in O ( n polylog n + k log k ) time. This is within a polylogarithmic factor of O ( n + k log k ), i.e., the best known complexity for equi-joins, and even of O ( n + k ), i.e., the time it takes to look at the input and return k answers in any order. Our guarantees extend to join queries with selections and many types of projections (namely those called "free-connex" queries and those that use bag semantics). Remarkably, they hold even when the number of join results is n ℓ for a join of ℓ relations. The key ingredient is a novel O ( n polylog n )-size factorized representation of the query output , which is constructed on-the-fly for a given query and database. In addition to providing the first nontrivial theoretical guarantees beyond equi-joins, we show in an experimental study that our ranked-enumeration approach is also memory-efficient and fast in practice, beating the running time of state-of-the-art database systems by orders of magnitude.

scholarly journals Improved N-Best Extraction with an Evaluation on Language Data

2021 ◽  
pp. 1-35
Author(s):  
Johanna Björklund ◽  
Frank Drewes ◽  
Anna Jonsson
Keyword(s):  
Language Processing ◽  
State Of The Art ◽  
Search Space ◽  
Data Sets ◽  
Weighted Tree ◽  
Original Algorithm ◽  
Software Toolkit ◽  
Minimal Weight ◽  
Language Data ◽  
Memory Efficient

Abstract We show that a previously proposed algorithm for the N-best trees problem can be made more efficient by changing how it arranges and explores the search space. Given an integer N and a weighted tree automaton (wta) M over the tropical semiring, the algorithm computes N trees of minimal weight with respect to M. Compared to the original algorithm, the modifications increase the laziness of the evaluation strategy, which makes the new algorithm asymptotically more efficient than its predecessor. The algorithm is implemented in the software Betty, and compared to the state-of-the-art algorithm for extracting the N best runs, implemented in the software toolkit Tiburon. The data sets used in the experiments are wtas resulting from real-world natural language processing tasks, as well as artificially created wtas with varying degrees of nondeterminism. We find that Betty outperforms Tiburon on all tested data sets with respect to running time, while Tiburon seems to be the more memory-efficient choice.

scholarly journals Numerical Characterization of the Load Distribution in Ball Screws

2021 ◽  
Author(s):  
Luca Sangalli ◽  
Aitor Oyanguren ◽  
Jon Larrañaga ◽  
Aitor Arana ◽  
Mikel Izquierdo ◽  
...  
Keyword(s):  
Load Distribution ◽  
State Of The Art ◽  
Ball Screw ◽  
Numerical Characterization ◽  
Design Variables ◽  
Ball Contact ◽  
Machine Elements ◽  
Characteristic Features ◽  
Variable Analysis

Abstract Load distribution in ball screws is a representation of the ball contact stress, and it is fundamental to understanding the behavior of these machine elements. This work aims to conduct a multi-variable analysis of the load distribution in ball screws. For this purpose, a numerical tool is developed for the generation and calculation of ball screw FEM models, which has been validated against the state of the art. Many different design variables are studied in order to obtain a general characterization of the morphology of the load distribution in ball screws. The two most characteristic features, the non-uniformity at a local and global level are identified, along with as the possible causes of their appearance and the consequences that they may cause.

scholarly journals C++ Modules in ROOT and Beyond

2020 ◽  
Vol 245 ◽  
pp. 05011
Author(s):  
Vassil Vassilev ◽  
David Lange ◽  
Malik Shahzad Muzaffar ◽  
Mircho Rodozov ◽  
Oksana Shadura ◽  
...  
Keyword(s):  
User Feedback ◽  
Memory Usage ◽  
Improve Performance ◽  
The Road ◽  
Road Map ◽  
Current State ◽  
Performance Optimizations ◽  
Memory Footprint ◽  
Intensive Development

C++ Modules, one of the new features of C++20, aim to fix the long-standing build scalability problems in the language. They provide an IOefficient, on-disk representation capable to reduce build times and peak memory usage. ROOT already employs the C++ modules technology in its dictionary system to improve performance and reduce the memory footprint. ROOT with C++ Modules was released as a technology preview in fall 2018, after intensive development during the previous few years. The current state is ready for production, however, there is still room for performance optimizations. In this talk, we show the road map for making this technology enabled by default in ROOT. We demonstrate a global module indexing optimization which allows reducing the memory footprint dramatically for many workflows. We will report user feedback on the migration to ROOT with C++ Modules.

scholarly journals Voxelisation Algorithms and Data Structures: A Review

Sensors ◽  
2021 ◽  
Vol 21 (24) ◽  
pp. 8241
Author(s):  
Mitko Aleksandrov ◽  
Sisi Zlatanova ◽  
David J. Heslop
Keyword(s):  
Data Structure ◽  
Data Structures ◽  
State Of The Art ◽  
Algorithms And Data Structures ◽  
3D Objects ◽  
Digital Representations ◽  
Memory Footprint ◽  
Voxel Data ◽  
2D And 3D ◽  
Geometric Primitives

Voxel-based data structures, algorithms, frameworks, and interfaces have been used in computer graphics and many other applications for decades. There is a general necessity to seek adequate digital representations, such as voxels, that would secure unified data structures, multi-resolution options, robust validation procedures and flexible algorithms for different 3D tasks. In this review, we evaluate the most common properties and algorithms for voxelisation of 2D and 3D objects. Thus, many voxelisation algorithms and their characteristics are presented targeting points, lines, triangles, surfaces and solids as geometric primitives. For lines, we identify three groups of algorithms, where the first two achieve different voxelisation connectivity, while the third one presents voxelisation of curves. We can say that surface voxelisation is a more desired voxelisation type compared to solid voxelisation, as it can be achieved faster and requires less memory if voxels are stored in a sparse way. At the same time, we evaluate in the paper the available voxel data structures. We split all data structures into static and dynamic grids considering the frequency to update a data structure. Static grids are dominated by SVO-based data structures focusing on memory footprint reduction and attributes preservation, where SVDAG and SSVDAG are the most advanced methods. The state-of-the-art dynamic voxel data structure is NanoVDB which is superior to the rest in terms of speed as well as support for out-of-core processing and data management, which is the key to handling large dynamically changing scenes. Overall, we can say that this is the first review evaluating the available voxelisation algorithms for different geometric primitives as well as voxel data structures.

Sign in / Sign up

Export Citation Format

Share Document