Systematic comparison of ranking aggregation methods for gene lists in experimental results

A common experimental output in biomedical science is a list of genes implicated in a given biological process or disease. The results of a group of studies answering the same, or similar, questions can be combined by meta-analysis to find a consensus or a more reliable answer. Ranking aggregation methods can be used to combine gene lists from various sources in meta-analyses. Evaluating a ranking aggregation method on a specific type of dataset before using it is required to support the reliability of the result since the property of a dataset can influence the performance of an algorithm. Evaluation of aggregation methods is usually based on a simulated database especially for the algorithms designed for gene lists because of the lack of a known truth for real data. However, simulated datasets tend to be too small compared to experimental data and neglect key features, including heterogeneity of quality, relevance and the inclusion of unranked lists. In this study, a group of existing methods and their variations which are suitable for meta-analysis of gene lists are compared using simulated and real data. Simulated data was used to explore the performance of the aggregation methods as a function of emulating the common scenarios of real genomics data, with various heterogeneity of quality, noise level, and a mix of unranked and ranked data using 20000 possible entities. In addition to the evaluation with simulated data, a comparison using real genomic data on the SARS-CoV-2 virus, cancer (NSCLC), and bacteria (macrophage apoptosis) was performed. We summarise our evaluation results in terms of a simple flowchart to select a ranking aggregation method for genomics data.

Proportional ranking in primary elections: A case study

Many democratic political parties hold primary elections, which nicely reflects their democratic nature and promote, among other things, the democratic value of inclusiveness. However, the methods currently used for holding such primary elections may not be the most suitable, especially if some form of proportional ranking is desired. In this paper, we compare different algorithmic methods for holding primaries (i.e., different aggregation methods for voters’ ballots) by evaluating the degree of proportional ranking that is achieved by each of them using real-world data. In particular, we compare six different algorithms by analyzing real-world data from a recent primary election conducted by the Israeli Democratit party. Technically, we analyze unique voter data and evaluate the proportionality achieved by means of cluster analysis, aiming at pinpointing the representation that is granted to different voter groups under each of the algorithmic methods considered. Our finding suggest that, contrary to the most-prominent primaries algorithm used (i.e., Approval), other methods such as Sequential Proportional Approval or Phragmen can bring about better proportional ranking and thus may be better suited for primary elections in practice.

Improving On-Line Debates by Aggregating Citizen Support1

Currently, Digital Democracy is gaining momentum thanks to online participation platforms, which have emerged as innovative tools that enable citizens to participate in decision making processes. Through these tools, participants can issue proposals and engage into debates by both stating arguments in favour or against and/or by supporting other people’s arguments. In this paper we propose a new support aggregation method derived from the combination of two complementary aggregation methods previously introduced. Additionally, we propose a resilience metric for measuring the quality of the aggregated opinion. We apply our contributions to debates conducted in the Decidim participatory platform.

Mathematically aggregating experts’ predictions of possible futures

Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the “best” final prediction. When experts’ performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts’ estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.

The Efficiency of Aggregation Methods in Ensemble Filter Feature Selection Models

Ensemble feature selection is recommended as it proves to produce a more stable subset of features and a better classification accuracy when compared to the individual feature selection methods. In this approach, the output of feature selection methods, called base selectors, are combined using some aggregation methods. For filter feature selection methods, a list aggregation method is needed to aggregate output ranked lists into a single list, and since many list aggregation methods have been proposed the decision on which method to use to build the optimum ensemble model is a de facto question. In this paper, we investigate the efficiency of four aggregation methods, namely; Min, Median, Arithmetic Mean, and Geometric Mean. The performance of aggregation methods is evaluated using five datasets from different scientific fields with a variant number of instances and features. Besides, the classifies used in the evaluation are selected from three different classes, Trees, Rules, and Bayes. The experimental results show that 11 out of the 15 best performance results are corresponding to ensemble models. And out of the 11 best performance ensemble models, the most efficient aggregation methods are Median (5/11), followed by Arithmetic Mean (3/11) and Min (3/11). Also, results show that as the number of features increased, the efficient aggregation method changes from Min to Median to Arithmetic Mean. This may suggest that for a very high number of features the efficient aggregation method is the Arithmetic Mean. And generally, there is no aggregation method that is the best for all cases.

Sequential group recommendations based on satisfaction and disagreement scores

Recently, group recommendations have gained much attention. Nevertheless, most approaches consider only one round of recommendations. However, in a real-life scenario, it is expected that the history of previous recommendations is exploited to tailor the recommendations towards meeting the needs of the group members. Such history should include not only which items the system suggested, but also the reaction of the members to these items. This work introduces the problem of sequential group recommendations, by exploiting the concept of satisfaction and disagreement. Satisfaction describes how well the group received the suggested items. Disagreement describes the satisfaction bias among the group members. We utilize these concepts in three new aggregation methods, SDAA, SIAA and Average+, designed to address the specific challenges introduced by sequential group recommendations. We experimentally show the effectiveness of our methods using big real datasets for both stable and ephemeral groups.

Wellbeing index avoiding scaling and weights

Objective: Existing well-being measures differ in terms of number and format of items, factors being measured, aggregation methods, and are not comparable. A well-being measure involves combining n- number of indicators and quality of the measure depends on properties of combining procedures adopted. The paper proposes two assumption-free aggregation methods to satisfy the desired properties of an index Methods: The paper proposes two indices of well-being in terms of cosine similarity and Geometric mean (GM) avoiding problems associated with scaling of raw data and choosing of weights. Empirical illustration is provided on application of the proposed measures. Results: The proposed indices give better admissibility of operations and satisfy properties like time-reversal test, formation of chain indices, computation of group mean and statistical tests for comparison across time and space. The preferred index can be constructed even for skewed longitudinal data and helps to reflect path of improvement registered by a country/region over time. Conclusions: The index based on GM is preferred due to wider application areas. The index can further be used for classification of countries, sub-groups and even individuals with morbidity in terms of overall wellbeing values. Future studies suggested.