Measuring and Optimizing Design Variety Using Herfindahl Index

2019 ◽  
Author(s):  
Faez Ahmed ◽  
Sharath Kumar Ramachandran ◽  
Mark Fuge ◽  
Sam Hunter ◽  
Scarlett Miller
Keyword(s):  
Ground Truth ◽  
Modular Function ◽  
Pairwise Comparisons ◽  
Herfindahl Index ◽  
New Variety ◽  
Good Design ◽  
Practical Procedure ◽  
Optimizing Design ◽  
Shed Light ◽  
Higher Sensitivity

Abstract In this paper, we propose a new design variety metric based on the Herfindahl index. We also propose a practical procedure for comparing variety metrics via the construction of ground truth datasets from pairwise comparisons by experts. Using two new datasets, we show that this new variety measure aligns with human ratings more than some existing and commonly used tree-based metrics. This metric also has three main advantages over existing metrics: a) It is a super-modular function, which enables us to optimize design variety using a polynomial time greedy algorithm. b) The parametric nature of this metric allows us to fit the metric to better represent variety for new domains. c) It has higher sensitivity in distinguishing between variety of sets of randomly selected designs than existing methods. Overall, our results shed light on some qualities that good design variety metrics should possess and the non-trivial challenges associated with collecting the data needed to measure those qualities.

Design Variety Measurement Using Sharma–Mittal Entropy

2020 ◽  
Vol 143 (6) ◽  
Author(s):  
Faez Ahmed ◽  
Sharath Kumar Ramachandran ◽  
Mark Fuge ◽  
Sam Hunter ◽  
Scarlett Miller
Keyword(s):  
Greedy Algorithm ◽  
Design Space ◽  
Ground Truth ◽  
Submodular Function ◽  
Pairwise Comparisons ◽  
Good Design ◽  
Measure Design ◽  
Multiple Metrics ◽  
Shed Light ◽  
Higher Sensitivity

Abstract Design variety metrics measure how much a design space is explored. This article proposes that a generalized class of entropy metrics based on Sharma–Mittal entropy offers advantages over existing methods to measure design variety. We show that an exemplar metric from Sharma–Mittal entropy, namely, the Herfindahl–Hirschman index for design (HHID) has the following desirable advantages over existing metrics: (a) more accuracy: it better aligns with human ratings compared to existing and commonly used tree-based metrics for two new datasets; (b) higher sensitivity: it has higher sensitivity compared to existing methods when distinguishing between the variety of sets; (c) allows efficient optimization: it is a submodular function, which enables one to optimize design variety using a polynomial time greedy algorithm; and (d) generalizes to multiple metrics: many existing metrics can be derived by changing the parameters of this metric, which allows a researcher to fit the metric to better represent variety for new domains. This article also contributes a procedure for comparing metrics used to measure variety via constructing ground truth datasets from pairwise comparisons. Overall, our results shed light on some qualities that good design variety metrics should possess and the nontrivial challenges associated with collecting the data needed to measure those qualities.

Interpreting Idea Maps: Pairwise Comparisons Reveal What Makes Ideas Novel

2018 ◽  
Vol 141 (2) ◽  
Author(s):  
Faez Ahmed ◽  
Sharath Kumar Ramachandran ◽  
Mark Fuge ◽  
Samuel Hunter ◽  
Scarlett Miller
Keyword(s):  
Ground Truth ◽  
Spatial Arrangement ◽  
Human Judgment ◽  
Physical Maps ◽  
Design Idea ◽  
Design Ideas ◽  
Mental Connections ◽  
Insight Into ◽  
Inherent Part ◽  
Shed Light

Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets to score ideas on their uniqueness. However, their decisions about novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two-dimensional (2D) idea maps. We derive these maps by asking participants for simple similarity comparisons of the form “Is idea A more similar to idea B or to idea C?” We show that these maps give insight into the relationships between ideas and help understand the design domain. We also propose that novel ideas can be identified by finding outliers on these idea maps. To demonstrate our method, we conduct experimental evaluations on two datasets—colored polygons (known answer) and milk frother sketches (unknown answer). We show that idea maps shed light on factors considered by participants in judging idea similarity and the maps are robust to noisy ratings. We also compare physical maps made by participants on a white-board to their computationally generated idea maps to compare how people think about spatial arrangement of design items. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods.

Unpacking Subjective Creativity Ratings: Using Embeddings to Explain and Measure Idea Novelty

2018 ◽  
Author(s):  
Faez Ahmed ◽  
Mark Fuge ◽  
Sam Hunter ◽  
Scarlett Miller
Keyword(s):  
Computational Methods ◽  
Ground Truth ◽  
Two Dimensional ◽  
Human Judgment ◽  
Design Idea ◽  
Design Ideas ◽  
Mental Connections ◽  
Insight Into ◽  
Inherent Part ◽  
Shed Light

Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets and scoring ideas on their uniqueness. However, their decisions on novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two dimensional idea maps. We derive these maps by asking humans for simple similarity comparisons of the form “Is idea A more similar to idea B or to idea C?” We show that these maps give insight into the relationships between ideas and help understand the domain. We also propose that the novelty of ideas can be estimated by measuring how far items are on these maps. We demonstrate our methodology through the experimental evaluations on two datasets of colored polygons (known answer) and milk frothers (unknown answer) sketches. We show that these maps shed light on factors considered by raters in judging idea similarity. We also show how maps change when less data is available or false/noisy ratings are provided. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods.

scholarly journals Semantic fieldwork and lexical universals

2014 ◽  
Vol 38 (1) ◽  
pp. 80-127 ◽  
Author(s):  
Cliff Goddard ◽  
Anna Wierzbicka
Keyword(s):  
Semantic Analysis ◽  
Endangered Languages ◽  
Methodological Issues ◽  
Practical Procedure ◽  
Semantic Communication ◽  
Shed Light

The main goal of paper is to show how NSM findings about lexical universals (semantic primes) can be applied to semantic analysis in little-described languages. It is argued that using lexical universals as a vocabulary for semantic analysis allows one to formulate meaning descriptions that are rigorous, cognitively authentic, maximally translatable, and free from Anglocentrism. A second goal is to shed light on methodological issues in semantic fieldwork by interrogating some controversial claims about the Dalabon and Pirahã languages. We argue that reductive paraphrase into lexical universals provides a practical procedure for arriving at coherent interpretations of unfamiliar lexical meanings. Other indigenous/endangered languages discussed include East Cree, Arrernte, Kayardild, Karuk, and Maori. We urge field linguists to take the NSM metalanguage, based on lexical universals, into the field with them, both as an aid to lexicogrammatical documentation and analysis and as a way to improve semantic communication with consultants.

scholarly journals Investigation on Non-Segmentation Based Algorithms for Microvasculature Quantification in OCTA Images

2021 ◽  
Vol 7 (2) ◽  
pp. 247-250
Author(s):  
Amr Abuzer ◽  
Ady Naber ◽  
Simon Hoffmann ◽  
Lucy Kessler ◽  
Ramin Khoramnia ◽  
...  
Keyword(s):  
Fractal Dimension ◽  
Imaging Modality ◽  
Ground Truth ◽  
Added Value ◽  
Data Set ◽  
Box Counting ◽  
Box Counting Dimension ◽  
Retinal Microvasculature ◽  
Higher Sensitivity

Abstract Optical Coherence Tomography Angiography (OCTA) is an imaging modality that provides threedimensional information of the retinal microvasculature and therefore promises early diagnosis and sufficient monitoring in ophthalmology. However, there is considerable variability between experts analysing this data. Measures for quantitative assessment of the vasculature need to be developed and established, such as fractal dimension. Fractal dimension can be used to assess the complexity of vessels and has been shown to be independently associated with neovascularization, a symptom of diseases such as diabetic retinopathy. This investigation assessed the performance of three fractal dimension algorithms: Box Counting Dimension (BCD), Information Dimension (ID), and Differential Box Counting (DBC). Two of those, BCD and ID, rely on previous vessel segmentation. Assessment of the added value or disturbance regarding the segmentation step is a second aim of this study. The investigation was performed on a data set composed of 9 in vivo human eyes. Since there is no ground truth available, the performance of the methods in differentiating the Superficial Vascular Complex (SVC) and Deep Vascular Complex (DVC) layers apart and the consistency of measurements of the same layer at different time-points were tested. The performance parameters were the ICC and the Mann- Whitney U tests. The three applied methods were suitable to tell the different layers apart and showed consistent values applied in the same slab. Within the consistency test, the non-segmentation-based method, DBC, was found to be less accurate, expressed in a lower ICC value, compared to its segmentation-based counterparts. This result is thought to be due to the DBC’s higher sensitivity when compared to the other methods. This higher sensitivity might help detect changes in the microvasculature, like neovascularization, but is also more likely prone to noise and artefacts.

scholarly journals Characterizing the Progression of Pulmonary Edema Severity: Can Pairwise Comparisons in Radiology Reports Help?

2022 ◽  
Author(s):  
Stephanie Hu ◽  
Steven Horng ◽  
Seth J. Berkowitz ◽  
Ruizhi Liao ◽  
Rahul G. Krishnan ◽  
...  
Keyword(s):  
Pulmonary Edema ◽  
Language Processing ◽  
Ground Truth ◽  
Pairwise Comparisons ◽  
Regular Expressions ◽  
Clinical Knowledge ◽  
Heart Failure Patients ◽  
Fine Grained ◽  
Radiology Reports ◽  
Severity Levels

Accurately assessing the severity of pulmonary edema is critical for making treatment decisions in congestive heart failure patients. However, the current scale for quantifying pulmonary edema based on chest radiographs does not have well-characterized severity levels, with substantial inter-radiologist disagreement. In this study, we investigate whether comparisons documented in radiology reports can provide accurate characterizations of pulmonary edema progression. We propose a rules-based natural language processing approach to assess the change in a patient's pulmonary edema status (e.g. better, worse, no change) by performing pairwise comparisons of consecutive radiology reports, using regular expressions and heuristics derived from clinical knowledge. Evaluated against ground-truth labels from expert radiologists, our labeler extracts comparisons describing the progression of pulmonary edema with 0.875 precision and 0.891 recall. We also demonstrate the potential utility of comparison labels in providing additional fine-grained information over noisier labels produced by models that directly estimate severity level.

scholarly journals Individual vs. Collaborative Methods of Crowdsourced Transcription

2019 ◽  
Vol Special Issue on Collecting,... ◽  
Author(s):  
Samantha Blickhan ◽  
Coleman Krawczyk ◽  
Daniel Hanson ◽  
Amy Boyer ◽  
Andrea Simenstad ◽  
...  
Keyword(s):  
Design Methodology ◽  
Research Question ◽  
Public Library ◽  
Ground Truth ◽  
Message Board ◽  
Quality Outcomes ◽  
Ground Truth Data ◽  
Project Outcomes ◽  
International Audience ◽  
Shed Light

International audience While online crowdsourced text transcription projects have proliferated in the last decade, there is a need within the broader field to understand differences in project outcomes as they relate to task design, as well as to experiment with different models of online crowdsourced transcription that have not yet been explored. The experiment discussed in this paper involves the evaluation of newly-built tools on the Zooniverse.org crowdsourcing platform, attempting to answer the research question: "Does the current Zooniverse methodology of multiple independent transcribers and aggregation of results render higher-quality outcomes than allowing volunteers to see previous transcriptions and/or markings by other users? How does each methodology impact the quality and depth of analysis and participation?" To answer these questions, the Zooniverse team ran an A/B experiment on the project Anti-Slavery Manuscripts at the Boston Public Library. This paper will share results of this study, and also describe the process of designing the experiment and the metrics used to evaluate each transcription method. These include the comparison of aggregate transcription results with ground truth data; evaluation of annotation methods; the time it took for volunteers to complete transcribing each dataset; and the level of engagement with other project elements such as posting on the message board or reading supporting documentation. Particular focus will be given to the (at times) competing goals of data quality, efficiency, volunteer engagement, and user retention, all of which are of high importance for projects that focus on data from galleries, libraries, archives and museums. Ultimately, this paper aims to provide a model for impactful, intentional design and study of online crowdsourcing transcription methods, as well as shed light on the associations between project design, methodology and outcomes.

scholarly journals Diversify your workflow! - An inconvenient suggestion to analyze spike data from intracranial recordings

2021 ◽  
Author(s):  
Şükrü Okkesim ◽  
Shavika Rastogi ◽  
Olaf Christ ◽  
Peter Hubka ◽  
Nicole Rosskothen-Kuhl ◽  
...  
Keyword(s):  
Action Potentials ◽  
Ground Truth ◽  
Initial Step ◽  
Data Set ◽  
Neuroscience Research ◽  
Different Types ◽  
Important Challenge ◽  
Detection Schemes ◽  
Intracranial Recordings ◽  
Shed Light

AbstractAn important challenge of neuroscience research and future brain machine interfacing is the reliable assignment of spikes to putative neurons. By means of extracellular recordings, researchers try to match different types action potentials with their putative neuronal source and timing. Unfortunately, this procedure is by far not standardized and reliable, leading to many different suggestions and as many differing results. It appears that sharing of data is thus hampered by different processing pipelines in different labs, thus playing along the reproducibility crisis in neurosciences. To systematically shed light on this issue, we present preliminary results of several easy event detection schemes on one data set, meant to illustrate the inconsistencies arising from different processing pipelines already in its initial step. The results indicate that thresholding choices alter findings due to a lack of a ground truth for spike sorting. We suggest to increase reliability in findings by only accepting and further processing events accepted by more than one processing pipeline.

A New AID for Interpolating and Assessing Collision Strengths and Rate Coefficients

1988 ◽  
Vol 102 ◽  
pp. 107-110
Author(s):  
A. Burgess ◽  
H.E. Mason ◽  
J.A. Tully
Keyword(s):  
Significant Loss ◽  
Impact Excitation ◽  
Rate Coefficients ◽  
Electron Impact Excitation ◽  
Graphical Display ◽  
Practical Procedure ◽  
Positive Ions ◽  
Allowed Transition ◽  
Computational Errors ◽  
Existing Data

AbstractA new way of critically assessing and compacting data for electron impact excitation of positive ions is proposed. This method allows one (i) to detect possible printing and computational errors in the published tables, (ii) to interpolate and extrapolate the existing data as a function of energy or temperature, and (iii) to simplify considerably the storage and transfer of data without significant loss of information. Theoretical or experimental collision strengths Ω(E) are scaled and then plotted as functions of the colliding electron energy, the entire range of which is conveniently mapped onto the interval (0,1). For a given transition the scaled Ω can be accurately represented - usually to within a fraction of a percent - by a 5 point least squares spline. Further details are given in (2). Similar techniques enable thermally averaged collision strengths upsilon (T) to be obtained at arbitrary temperatures in the interval 0 < T < ∞. Application of the method is possible by means of an interactive program with graphical display (2). To illustrate this practical procedure we use the program to treat Ω for the optically allowed transition 2s → 2p in ArXVI.

Elucidating cyclic AMP signaling in subcellular domains with optogenetic tools and fluorescent biosensors

2019 ◽  
Vol 47 (6) ◽  
pp. 1733-1747 ◽  
Author(s):  
Christina Klausen ◽  
Fabian Kaiser ◽  
Birthe Stüven ◽  
Jan N. Hansen ◽  
Dagmar Wachten
Keyword(s):  
Cyclic Amp ◽  
Adenosine Monophosphate ◽  
Adenylyl Cyclases ◽  
Temporal Precision ◽  
Fluorescent Biosensors ◽  
Subcellular Domains ◽  
Spatio Temporal ◽  
Hydrolysis Of ◽  
Shed Light ◽  
Cyclic Amp Signaling

The second messenger 3′,5′-cyclic nucleoside adenosine monophosphate (cAMP) plays a key role in signal transduction across prokaryotes and eukaryotes. Cyclic AMP signaling is compartmentalized into microdomains to fulfil specific functions. To define the function of cAMP within these microdomains, signaling needs to be analyzed with spatio-temporal precision. To this end, optogenetic approaches and genetically encoded fluorescent biosensors are particularly well suited. Synthesis and hydrolysis of cAMP can be directly manipulated by photoactivated adenylyl cyclases (PACs) and light-regulated phosphodiesterases (PDEs), respectively. In addition, many biosensors have been designed to spatially and temporarily resolve cAMP dynamics in the cell. This review provides an overview about optogenetic tools and biosensors to shed light on the subcellular organization of cAMP signaling.

Sign in / Sign up

Export Citation Format

Share Document