scholarly journals Fast and compact matching statistics analytics

2021 ◽  
Author(s):  
Fabio Cunial ◽  
Olgert Denas ◽  
Djamal Belazzougui
Keyword(s):  
Shared Memory ◽  
Data Structures ◽  
Large Scale ◽  
State Of The Art ◽  
Structural Rearrangements ◽  
Compression Scheme ◽  
Local Statistics ◽  
Compact Representations ◽  
Long Reads ◽  
New Applications

Fast, lightweight methods for comparing the sequence of ever larger assembled genomes from ever growing databases are increasingly needed in the era of accurate long reads and pan-genome initiatives. Matching statistics is a popular method for computing whole-genome phylogenies and for detecting structural rearrangements between two genomes, since it is amenable to fast implementations that require a minimal setup of data structures. However, current implementations use a single core, take too much memory to represent the result, and do not provide efficient ways to analyze the output in order to explore local similarities between the sequences. We develop practical tools for computing matching statistics between large-scale strings, and for analyzing its values, faster and using less memory than the state of the art. Specifically, we design a parallel algorithm for shared-memory machines that computes matching statistics 30 times faster with 48 cores in the cases that are most difficult to parallelize. We design a lossy compression scheme that shrinks the matching statistics array to a bitvector that takes from 0.8 to 0.2 bits per character, depending on the dataset and on the value of a threshold, and that achieves 0.04 bits per character in some variants. And we provide efficient implementations of range-maximum and range-sum queries that take a few tens of milliseconds while operating on our compact representations, and that allow computing key local statistics about the similarity between two strings. Our toolkit makes construction, storage, and analysis of matching statistics arrays practical for multiple pairs of the largest genomes available today, possibly enabling new applications in comparative genomics.

scholarly journals Tackling the Challenges of FASTQ Referential Compression

2019 ◽  
Vol 13 ◽  
pp. 117793221882137
Author(s):  
Aníbal Guerra ◽  
Jaime Lotero ◽  
José Édinson Aedo ◽  
Sebastián Isaza
Keyword(s):  
Storage Capacity ◽  
State Of The Art ◽  
Added Value ◽  
Performance Tests ◽  
Memory Consumption ◽  
Compression Scheme ◽  
Capacity Limitations ◽  
Novel Approach ◽  
Long Reads ◽  
And Performance

The exponential growth of genomic data has recently motivated the development of compression algorithms to tackle the storage capacity limitations in bioinformatics centers. Referential compressors could theoretically achieve a much higher compression than their non-referential counterparts; however, the latest tools have not been able to harness such potential yet. To reach such goal, an efficient encoding model to represent the differences between the input and the reference is needed. In this article, we introduce a novel approach for referential compression of FASTQ files. The core of our compression scheme consists of a referential compressor based on the combination of local alignments with binary encoding optimized for long reads. Here we present the algorithms and performance tests developed for our reads compression algorithm, named UdeACompress. Our compressor achieved the best results when compressing long reads and competitive compression ratios for shorter reads when compared to the best programs in the state of the art. As an added value, it also showed reasonable execution times and memory consumption, in comparison with similar tools.

scholarly journals cloudSPAdes: assembly of synthetic long reads using de Bruijn graphs

2019 ◽  
Vol 35 (14) ◽  
pp. i61-i70 ◽  
Author(s):  
Ivan Tolstoganov ◽  
Anton Bankevich ◽  
Zhoutao Chen ◽  
Pavel A Pevzner
Keyword(s):  
Narrow Range ◽  
State Of The Art ◽  
Supplementary Information ◽  
De Bruijn Graph ◽  
Hybrid Assembly ◽  
De Bruijn Graphs ◽  
Long Reads ◽  
Long Read ◽  
De Bruijn ◽  
New Applications

Abstract Motivation The recently developed barcoding-based synthetic long read (SLR) technologies have already found many applications in genome assembly and analysis. However, although some new barcoding protocols are emerging and the range of SLR applications is being expanded, the existing SLR assemblers are optimized for a narrow range of parameters and are not easily extendable to new barcoding technologies and new applications such as metagenomics or hybrid assembly. Results We describe the algorithmic challenge of the SLR assembly and present a cloudSPAdes algorithm for SLR assembly that is based on analyzing the de Bruijn graph of SLRs. We benchmarked cloudSPAdes across various barcoding technologies/applications and demonstrated that it improves on the state-of-the-art SLR assemblers in accuracy and speed. Availability and implementation Source code and installation manual for cloudSPAdes are available at https://github.com/ablab/spades/releases/tag/cloudspades-paper. Supplementary Information Supplementary data are available at Bioinformatics online.

scholarly journals Spacemake: processing and analysis of large-scale spatial transcriptomics data

2021 ◽  
Author(s):  
Tamas Ryszard Sztanka-Toth ◽  
Marvin Jens ◽  
Nikos Karaiskos ◽  
Nikolaus Rajewsky
Keyword(s):  
Large Scale ◽  
Modular Design ◽  
State Of The Art ◽  
Sequencing Data ◽  
Unified Framework ◽  
Tissue Sections ◽  
Long Reads ◽  
Rna Biology ◽  
Transcriptomics Data ◽  
Downstream Analysis

Spatial sequencing methods increasingly gain popularity within RNA biology studies. State-of-the-art techniques can read mRNA expression levels from tissue sections and at the same time register information about the original locations of the molecules in the tissue. The resulting datasets are processed and analyzed by accompanying software which, however, is incompatible across inputs from different technologies. Here, we present spacemake, a modular, robust and scalable spatial transcriptomics pipeline built in snakemake and python. Spacemake is designed to handle all major spatial transcriptomics datasets and can be readily configured to run on other technologies. It can process and analyze several samples in parallel, even if they stem from different experimental methods. Spacemake's unified framework enables reproducible data processing from raw sequencing data to automatically generated downstream analysis reports. Moreover, spacemake is built with a modular design and offers additional functionality such as sample merging, saturation analysis and analysis of long-reads as separate modules. Moreover, spacemake employs novoSpaRc to integrate spatial and single-cell transcriptomics data, resulting in increased gene counts for the spatial dataset. Spacemake is open-source, extendable and can be readily integrated with existing computational workflows.

scholarly journals Discrete Network Embedding

2018 ◽  
Author(s):  
Xiaobo Shen ◽  
Shirui Pan ◽  
Weiwei Liu ◽  
Yew-Soon Ong ◽  
Quan-Sen Sun
Keyword(s):  
Large Scale ◽  
State Of The Art ◽  
Ground Truth ◽  
Network Embedding ◽  
Unified Framework ◽  
Network Nodes ◽  
Compact Representations ◽  
Low Dimensional ◽  
Vector Representations ◽  
Embedding Methods

Network embedding aims to seek low-dimensional vector representations for network nodes, by preserving the network structure. The network embedding is typically represented in continuous vector, which imposes formidable challenges in storage and computation costs, particularly in large-scale applications. To address the issue, this paper proposes a novel discrete network embedding (DNE) for more compact representations. In particular, DNE learns short binary codes to represent each node. The Hamming similarity between two binary embeddings is then employed to well approximate the ground-truth similarity. A novel discrete multi-class classifier is also developed to expedite classification. Moreover, we propose to jointly learn the discrete embedding and classifier within a unified framework to improve the compactness and discrimination of network embedding. Extensive experiments on node classification consistently demonstrate that DNE exhibits lower storage and computational complexity than state-of-the-art network embedding methods, while obtains competitive classification results.

scholarly journals Documentary data and the study of past droughts: a global state of the art

2018 ◽  
Vol 14 (12) ◽  
pp. 1915-1960 ◽  
Author(s):  
Rudolf Brázdil ◽  
Andrea Kiss ◽  
Jürg Luterbacher ◽  
David J. Nash ◽  
Ladislava Řezníčková
Keyword(s):  
Large Scale ◽  
State Of The Art ◽  
Drought Indices ◽  
Documentary Evidence ◽  
Climatic Trends ◽  
Instrumental Observations ◽  
Spatio Temporal ◽  
Epigraphic Evidence ◽  
Administrative Evidence

Abstract. The use of documentary evidence to investigate past climatic trends and events has become a recognised approach in recent decades. This contribution presents the state of the art in its application to droughts. The range of documentary evidence is very wide, including general annals, chronicles, memoirs and diaries kept by missionaries, travellers and those specifically interested in the weather; records kept by administrators tasked with keeping accounts and other financial and economic records; legal-administrative evidence; religious sources; letters; songs; newspapers and journals; pictographic evidence; chronograms; epigraphic evidence; early instrumental observations; society commentaries; and compilations and books. These are available from many parts of the world. This variety of documentary information is evaluated with respect to the reconstruction of hydroclimatic conditions (precipitation, drought frequency and drought indices). Documentary-based drought reconstructions are then addressed in terms of long-term spatio-temporal fluctuations, major drought events, relationships with external forcing and large-scale climate drivers, socio-economic impacts and human responses. Documentary-based drought series are also considered from the viewpoint of spatio-temporal variability for certain continents, and their employment together with hydroclimate reconstructions from other proxies (in particular tree rings) is discussed. Finally, conclusions are drawn, and challenges for the future use of documentary evidence in the study of droughts are presented.

scholarly journals VIPPrint: Validating Synthetic Image Detection and Source Linking Methods on a Large Scale Dataset of Printed Documents

2021 ◽  
Vol 7 (3) ◽  
pp. 50
Author(s):  
Anselmo Ferreira ◽  
Ehsan Nowroozi ◽  
Mauro Barni
Keyword(s):  
Large Scale ◽  
State Of The Art ◽  
Child Pornography ◽  
Forensic Analysis ◽  
Synthetic Image ◽  
Image Detection ◽  
Face Images ◽  
Large Scale Dataset ◽  
Scanned Images ◽  
Analysis Of The Images

The possibility of carrying out a meaningful forensic analysis on printed and scanned images plays a major role in many applications. First of all, printed documents are often associated with criminal activities, such as terrorist plans, child pornography, and even fake packages. Additionally, printing and scanning can be used to hide the traces of image manipulation or the synthetic nature of images, since the artifacts commonly found in manipulated and synthetic images are gone after the images are printed and scanned. A problem hindering research in this area is the lack of large scale reference datasets to be used for algorithm development and benchmarking. Motivated by this issue, we present a new dataset composed of a large number of synthetic and natural printed face images. To highlight the difficulties associated with the analysis of the images of the dataset, we carried out an extensive set of experiments comparing several printer attribution methods. We also verified that state-of-the-art methods to distinguish natural and synthetic face images fail when applied to print and scanned images. We envision that the availability of the new dataset and the preliminary experiments we carried out will motivate and facilitate further research in this area.

scholarly journals SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks

2021 ◽  
Vol 40 (3) ◽  
pp. 1-13
Author(s):  
Lumin Yang ◽  
Jiajie Zhuang ◽  
Hongbo Fu ◽  
Xiangzhi Wei ◽  
Kun Zhou ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Large Scale ◽  
State Of The Art ◽  
Semantic Segmentation ◽  
Structure Information ◽  
Graph Neural Networks ◽  
Node Labels ◽  
Point Level

We introduce SketchGNN , a convolutional graph neural network for semantic segmentation and labeling of freehand vector sketches. We treat an input stroke-based sketch as a graph with nodes representing the sampled points along input strokes and edges encoding the stroke structure information. To predict the per-node labels, our SketchGNN uses graph convolution and a static-dynamic branching network architecture to extract the features at three levels, i.e., point-level, stroke-level, and sketch-level. SketchGNN significantly improves the accuracy of the state-of-the-art methods for semantic sketch segmentation (by 11.2% in the pixel-based metric and 18.2% in the component-based metric over a large-scale challenging SPG dataset) and has magnitudes fewer parameters than both image-based and sequence-based methods.

scholarly journals ShadingNet: Image Intrinsics by Fine-Grained Shading Decomposition

2021 ◽  
Author(s):  
Anil S. Baslamisli ◽  
Partha Das ◽  
Hoang-An Le ◽  
Sezer Karaoglu ◽  
Theo Gevers
Keyword(s):  
Neural Network ◽  
Large Scale ◽  
State Of The Art ◽  
Image Decomposition ◽  
Natural Environments ◽  
Decomposition Algorithms ◽  
Ambient Light ◽  
Fine Grained ◽  
Large Scale Dataset ◽  
Direct Illumination

AbstractIn general, intrinsic image decomposition algorithms interpret shading as one unified component including all photometric effects. As shading transitions are generally smoother than reflectance (albedo) changes, these methods may fail in distinguishing strong photometric effects from reflectance variations. Therefore, in this paper, we propose to decompose the shading component into direct (illumination) and indirect shading (ambient light and shadows) subcomponents. The aim is to distinguish strong photometric effects from reflectance variations. An end-to-end deep convolutional neural network (ShadingNet) is proposed that operates in a fine-to-coarse manner with a specialized fusion and refinement unit exploiting the fine-grained shading model. It is designed to learn specific reflectance cues separated from specific photometric effects to analyze the disentanglement capability. A large-scale dataset of scene-level synthetic images of outdoor natural environments is provided with fine-grained intrinsic image ground-truths. Large scale experiments show that our approach using fine-grained shading decompositions outperforms state-of-the-art algorithms utilizing unified shading on NED, MPI Sintel, GTA V, IIW, MIT Intrinsic Images, 3DRMS and SRD datasets.

scholarly journals On the relation between mini-halos and AGN feedback in clusters of galaxies

2020 ◽  
Vol 499 (2) ◽  
pp. 2934-2958
Author(s):  
A Richard-Laferrière ◽  
J Hlavacek-Larrondo ◽  
R S Nemmen ◽  
C L Rhea ◽  
G B Taylor ◽  
...  
Keyword(s):  
Large Scale ◽  
State Of The Art ◽  
Vital Role ◽  
Clusters Of Galaxies ◽  
Relativistic Jets ◽  
X Ray ◽  
Core Cluster ◽  
Central Galaxy ◽  
Central Regions ◽  
Radio Power

ABSTRACT A variety of large-scale diffuse radio structures have been identified in many clusters with the advent of new state-of-the-art facilities in radio astronomy. Among these diffuse radio structures, radio mini-halos are found in the central regions of cool core clusters. Their origin is still unknown and they are challenging to discover; less than 30 have been published to date. Based on new VLA observations, we confirmed the mini-halo in the massive strong cool core cluster PKS 0745−191 (z = 0.1028) and discovered one in the massive cool core cluster MACS J1447.4+0827 (z = 0.3755). Furthermore, using a detailed analysis of all known mini-halos, we explore the relation between mini-halos and active galactic nucleus (AGN) feedback processes from the central galaxy. We find evidence of strong, previously unknown correlations between mini-halo radio power and X-ray cavity power, and between mini-halo and the central galaxy radio power related to the relativistic jets when spectrally decomposing the AGN radio emission into a component for past outbursts and one for ongoing accretion. Overall, our study indicates that mini-halos are directly connected to the central AGN in clusters, following previous suppositions. We hypothesize that AGN feedback may be one of the dominant mechanisms giving rise to mini-halos by injecting energy into the intra-cluster medium and reaccelerating an old population of particles, while sloshing motion may drive the overall shape of mini-halos inside cold fronts. AGN feedback may therefore not only play a vital role in offsetting cooling in cool core clusters, but may also play a fundamental role in re-energizing non-thermal particles in clusters.

scholarly journals Extrinsic Camera Calibration with Line-Laser Projection

Sensors ◽  
2021 ◽  
Vol 21 (4) ◽  
pp. 1091
Author(s):  
Izaak Van Crombrugge ◽  
Rudi Penne ◽  
Steve Vanlanduit
Keyword(s):  
Camera Calibration ◽  
Real World ◽  
Large Scale ◽  
State Of The Art ◽  
Bundle Adjustment ◽  
Field Of View ◽  
Extrinsic Calibration ◽  
Practical Procedure ◽  
Partial Overlap

Knowledge of precise camera poses is vital for multi-camera setups. Camera intrinsics can be obtained for each camera separately in lab conditions. For fixed multi-camera setups, the extrinsic calibration can only be done in situ. Usually, some markers are used, like checkerboards, requiring some level of overlap between cameras. In this work, we propose a method for cases with little or no overlap. Laser lines are projected on a plane (e.g., floor or wall) using a laser line projector. The pose of the plane and cameras is then optimized using bundle adjustment to match the lines seen by the cameras. To find the extrinsic calibration, only a partial overlap between the laser lines and the field of view of the cameras is needed. Real-world experiments were conducted both with and without overlapping fields of view, resulting in rotation errors below 0.5°. We show that the accuracy is comparable to other state-of-the-art methods while offering a more practical procedure. The method can also be used in large-scale applications and can be fully automated.

Sign in / Sign up

Export Citation Format

Share Document