scholarly journals Towards Automatic Data Format Transformations: Data Wrangling at Scale

2017 ◽  
pp. 36-48 ◽  
Author(s):  
Alex Bogatu ◽  
Norman W. Paton ◽  
Alvaro A. A. Fernandes
Keyword(s):  
Data Format ◽  
Automatic Data

scholarly journals Towards Automatic Data Format Transformations: Data Wrangling at Scale

2018 ◽  
Vol 62 (7) ◽  
pp. 1044-1060 ◽  
Author(s):  
Alex Bogatu ◽  
Norman W Paton ◽  
Alvaro A A Fernandes ◽  
Martin Koehler
Keyword(s):  
Analysis Data ◽  
Data Sources ◽  
Open Government ◽  
Tool Support ◽  
Data Format ◽  
Automatic Data ◽  
Open Government Data ◽  
Automatic Discovery ◽  
Numerous Data ◽  
Government Data

Abstract Data wrangling is the process whereby data are cleaned and integrated for analysis. Data wrangling, even with tool support, is typically a labour intensive process. One aspect of data wrangling involves carrying out format transformations on attribute values, for example so that names or phone numbers are represented consistently. Recent research has developed techniques for synthesizing format transformation programs from examples of the source and target representations. This is valuable, but still requires a user to provide suitable examples, something that may be challenging in applications in which there are huge datasets or numerous data sources. In this paper, we investigate the automatic discovery of examples that can be used to synthesize format transformation programs. In particular, we propose two approaches to identifying candidate data examples and validating the transformations that are synthesized from them. The approaches are evaluated empirically using datasets from open government data.

scholarly journals DEVELOPMENT OF AUTOMATIC DATA PROCESSING FOR BATAN’S HRPD AND FCD/TD USING PYTHON CODE

2020 ◽  
Vol 22 (3) ◽  
pp. 97
Author(s):  
Muzakkiy Putra Muhammad Akhir ◽  
Rina Kamila
Keyword(s):  
Neutron Diffraction ◽  
Data Processing ◽  
Development Project ◽  
Automatic Data Processing ◽  
Single Experiment ◽  
Data Format ◽  
Automatic Data ◽  
Crystalline Materials ◽  
Processing Step ◽  
Fully Automatic

High Resolution Powder Diffractometer (HRPD) and Four Circle Diffractometer/Texture Diffractometer (FCD/TD) are two BATAN-owned neutron diffractometers which have been fully operational since 1992. These are used to investigate structure and texture of crystalline materials, respectively. Before analyzing, the acquired raw neutron diffraction data should first be processed in a specific way to achieve the suitable data format required by the analysis software. This data processing step is a repetitive task for every single experiment which is previously done manually and very time-consuming. The purpose of this development project was to optimize this step to be fully automatic and executable by a code. This work was performed by means of Python code utilizing the array manipulation in re-arranging and re-formatting the raw data. The resulted Python codes were named as hrpd.py and fcdtd.py. These have been successfully done and validated, making data processing step easier, simpler, and significantly faster with only 20 seconds or less required.Keywords: HRPD, FCD/TD, Automatic Data Processing, Neutron Diffraction, Python

Instrumentation For Quantitative Microscopy

1977 ◽  
Vol 35 ◽  
pp. 206-209
Author(s):  
B. Ralph ◽  
A.R. Jones
Keyword(s):  
Volume Fraction ◽  
Three Dimensional ◽  
Two Dimensional ◽  
Automatic Data ◽  
Phase Volume Fraction ◽  
Statistical Confidence ◽  
Resultant Data ◽  
Automatic Data Collection ◽  
Third Dimension ◽  
Evolution Of Microstructure

In all fields of microscopy there is an increasing interest in the quantification of microstructure. This interest may stem from a desire to establish quality control parameters or may have a more fundamental requirement involving the derivation of parameters which partially or completely define the three dimensional nature of the microstructure. This latter categorey of study may arise from an interest in the evolution of microstructure or from a desire to generate detailed property/microstructure relationships. In the more fundamental studies some convolution of two-dimensional data into the third dimension (stereological analysis) will be necessary.In some cases the two-dimensional data may be acquired relatively easily without recourse to automatic data collection and further, it may prove possible to perform the data reduction and analysis relatively easily. In such cases the only recourse to machines may well be in establishing the statistical confidence of the resultant data. Such relatively straightforward studies tend to result from acquiring data on the whole assemblage of features making up the microstructure. In this field data mode, when parameters such as phase volume fraction, mean size etc. are sought, the main case for resorting to automation is in order to perform repetitive analyses since each analysis is relatively easily performed.

Data Screening Methods – Application to Differential Diagnosis in Pancreatic Pathology from Radiological Signs

1978 ◽  
Vol 17 (01) ◽  
pp. 36-40 ◽  
Author(s):  
J.-P. Durbec ◽  
Jaqueline Cornée ◽  
P. Berthezene
Keyword(s):  
Differential Diagnosis ◽  
Mutual Information ◽  
Data Processing ◽  
Screening Methods ◽  
Automatic Data ◽  
Chronic Calcifying Pancreatitis ◽  
Number Of Patients ◽  
Calcifying Pancreatitis ◽  
Pancreatic Pathology ◽  
Data Screening

The practice of systematic examinations in hospitals and the increasing development of automatic data processing permits the storing of a great deal of information about a large number of patients belonging to different diagnosis groups.To predict or to characterize these diagnosis groups some descriptors are particularly useful, others carry no information. Data screening based on the properties of mutual information and on the log cross products ratios in contingency tables is developed. The most useful descriptors are selected. For each one the characterized groups are specified.This approach has been performed on a set of binary (presence—absence) radiological variables. Four diagnoses groups are concerned: cancer of pancreas, chronic calcifying pancreatitis, non-calcifying pancreatitis and probable pancreatitis. Only twenty of the three hundred and forty initial radiological variables are selected. The presence of each corresponding sign is associated with one or more diagnosis groups.

Sign in / Sign up

Export Citation Format

Share Document