A Diffusion Approximation Theory of Momentum Stochastic Gradient Descent in Nonconvex Optimization

Momentum stochastic gradient descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning (e.g., training deep neural networks, variational Bayesian inference, etc.). Despite its empirical success, there is still a lack of theoretical understanding of convergence properties of MSGD. To fill this gap, we propose to analyze the algorithmic behavior of MSGD by diffusion approximations for nonconvex optimization problems with strict saddle points and isolated local optima. Our study shows that the momentum helps escape from saddle points but hurts the convergence within the neighborhood of optima (if without the step size annealing or momentum annealing). Our theoretical discovery partially corroborates the empirical success of MSGD in training deep neural networks.

Generative Adversarial Neural Architecture Search

Despite the empirical success of neural architecture search (NAS) in deep learning applications, the optimality, reproducibility and cost of NAS schemes remain hard to assess. In this paper, we propose Generative Adversarial NAS (GA-NAS) with theoretically provable convergence guarantees, promoting stability and reproducibility in neural architecture search. Inspired by importance sampling, GA-NAS iteratively fits a generator to previously discovered top architectures, thus increasingly focusing on important parts of a large search space. Furthermore, we propose an efficient adversarial learning approach, where the generator is trained by reinforcement learning based on rewards provided by a discriminator, thus being able to explore the search space without evaluating a large number of architectures. Extensive experiments show that GA-NAS beats the best published results under several cases on three public NAS benchmarks. In the meantime, GA-NAS can handle ad-hoc search constraints and search spaces. We show that GA-NAS can be used to improve already optimized baselines found by other NAS methods, including EfficientNet and ProxylessNAS, in terms of ImageNet accuracy or the number of parameters, in their original search space.

Newton and Spinoza

This chapter investigates several arguments against Spinoza’s philosophy that were developed by Henry More, Samuel Clarke, and Colin Maclaurin. In the arguments More, Clarke, and Maclaurin aim to establish the existence of an immaterial and intelligent God precisely by showing that Spinoza does not have the resources to adequately explain the origin of motion. Attending to these criticisms grants us a deeper appreciation for how the authority derived from the empirical success of Newton's enterprise was used to settle debates within philosophy. The arguments by More and Clarke especially help to discern the anti‐Spinozism that can be detected in Newton's General Scholium (1713). Ultimately, the Newtonian criticisms of Spinoza offer us a more nuanced view of the problems that plague Spinoza's philosophy, and they also challenge the idea that Spinoza seamlessly fits into a progressive narrative about the scientific revolution.

Deep Graph Mapper: Seeing Graphs Through the Neural Lens

Graph summarization has received much attention lately, with various works tackling the challenge of defining pooling operators on data regions with arbitrary structures. These contrast the grid-like ones encountered in image inputs, where techniques such as max-pooling have been enough to show empirical success. In this work, we merge the Mapper algorithm with the expressive power of graph neural networks to produce topologically grounded graph summaries. We demonstrate the suitability of Mapper as a topological framework for graph pooling by proving that Mapper is a generalization of pooling methods based on soft cluster assignments. Building upon this, we show how easy it is to design novel pooling algorithms that obtain competitive results with other state-of-the-art methods. Additionally, we use our method to produce GNN-aided visualisations of attributed complex networks.

What do post-disaster reconstruction project success indicators look like? End-user’s perspectives

Purpose This paper aims to investigate the emergence of new success measures for buildings and infrastructure post-disaster reconstruction projects, beyond the traditional ”iron triangle”, which have gained prominence with the increased involvement of clients and end-users in these projects. Consequently, the industry is obliged to reconsider the critical factors regarding what constitutes a successful outcome from the perspectives of these stakeholders. Design/methodology/approach Data was gathered from end-users in four Caribbean islands using a questionnaire survey on eight empirical success indicators obtained from an extensive systematic literature review. To elicit a ranking and correlations amongst the end-user’ perspectives on the indicators, factor analysis and structural equation modelling techniques (SEM) were conducted. Findings The factor analysis found “safety” to be the most important empirical success measure, while “change” ranked the least important. Correlation analysis using SEM identified two new composite indicators, namely, “competence” with delivering timely and quality environmentally friendly and sustainable projects and “adaptability” in ensuring project objectives reflect beneficiaries’ expectations amidst internal and external influences, to be critical of end-users’ measurement indicators that describe their assessment mechanism. Measurement and structural models validated “safety” and “satisfaction” to be the highest loading variables in the two composites, respectively. Research limitations/implications The research focussed on findings in English language articles; therefore, any claim to a complete list of indicators from the literature can be amiss. Practical implications Results confirm the traditional “iron triangle” of time, cost and quality to be limited in assessing reconstruction project outcomes and the views and expectations of the potential beneficiaries need to be factored in the planning, design, execution and post-handover stages in all reconstruction projects. Originality/value This paper was very specific in its attempt to investigate new success indicators for reconstruction project outcomes, aiming to assist with developing comprehensive project objectives that resonate with all stakeholder groups.

Newton’s Scientific Method

One central feature of Newton’s methodology is the use of theory-mediated measurements to make empirical phenomena carry information about causal dependencies. Accurate measurements of theoretical parameters by phenomena is a conception of empirical success that goes beyond the restriction of empirical success to accurate prediction of phenomena in the hypothetico-deductive model of scientific inference. This is informed by Newton’s account of provisional acceptance of propositions gathered from phenomena in his Fourth Rule of reasoning. Another feature is to follow up the discovery of forces of nature by applying those forces to additional phenomena. This is illustrated by the sequence of refinements of the model for solar system motions as more and more dependencies corresponding to Newtonian interactions were taken into account. Given that his theory would recover these Newtonian causal dependencies, Einstein’s Mercury perihelion result made his theory of gravity more empirically successful than Newton’s.

Statistical Decision Properties of Imprecise Trials Assessing COVID-19 Drugs

As the COVID-19 pandemic progresses, researchers are reporting findings of randomized trials comparing standard care with care augmented by experimental drugs. The trials have small sample sizes, so estimates of treatment effects are imprecise. Seeing imprecision, clinicians reading research articles may find it difficult to decide when to treat patients with experimental drugs. Whatever decision criterion one uses, there is always some probability that random variation in trial outcomes will lead to prescribing sub-optimal treatments. A conventional practice when comparing standard care and an innovation is to choose the innovation only if the estimated treatment effect is positive and statistically significant. This practice defers to standard care as the status quo. To evaluate decision criteria, we use the concept of near optimality, which jointly considers the probability and magnitude of decision errors. An appealing decision criterion from this perspective is the empirical success rule, which chooses the treatment with the highest observed average patient outcome in the trial. Considering the design of recent and ongoing COVID-19 trials, we show that the empirical success rule yields treatment results that are much closer to optimal than those generated by prevailing decision criteria based on hypothesis tests.We have benefitted from the comments of Michael Gmeiner, Valentyn Litvin, Francesca Molinari, and John Mullahy. Tetenov has received funding from the Swiss National Science Foundation through grant number 100018-192580.