Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation

In this paper, we give a simple and efficient implementation of reverse-mode automatic differentiation, which both extends easily to higher-order functions, and has run time and memory consumption linear in the run time of the original program. In addition to a formal description of the translation, we also describe an implementation of this algorithm, and prove its correctness by means of a logical relations argument.

Non-zero Yarkovsky acceleration for near-Earth asteroid (99942) Apophis

The leading source of uncertainty to predict the orbital motion of asteroid (99942) Apophis is a non-gravitational acceleration arising from the anisotropic thermal re-emission of absorbed radiation, known as the Yarkovsky effect. Previous attempts to obtain this parameter from astrometry for this object have only yielded marginally small values, without ruling out a pure gravitational interaction. Here we present an independent estimation of the Yarkovsky effect based on optical and radar astrometry which includes observations obtained during 2021. Our numerical approach exploits automatic differentiation techniques. We find a non-zero Yarkovsky parameter, A2 = (−2.899 ± 0.025) × 10−14 au d−2, with induced semi-major axis drift of (−199.0 ± 1.5) m yr−1 for Apophis. Our results provide definite collision probability predictions for the close approaches in 2029, 2036, and 2068.

AIEADA 1.0: Efficient high-dimensional variational data assimilation with machine-learned reduced-order models

Data assimilation (DA) in the geophysical sciences remains the cornerstone of robust forecasts from numerical models. Indeed, DA plays a crucial role in the quality of numerical weather prediction, and is a crucial building block that has allowed dramatic improvements in weather forecasting over the past few decades. DA is commonly framed in a variational setting, where one solves an optimization problem within a Bayesian formulation using raw model forecasts as a prior, and observations as likelihood. This leads to a DA objective function that needs to be minimized, where the decision variables are the initial conditions specified to the model. In traditional DA, the forward model is numerically and computationally expensive. Here we replace the forward model with a low-dimensional, data-driven, and differentiable emulator. Consequently, gradients of our DA objective function with respect to the decision variables are obtained rapidly via automatic differentiation. We demonstrate our approach by performing an emulator-assisted DA forecast of geopotential height. Our results indicate that emulator-assisted DA is faster than traditional equation-based DA forecasts by four orders of magnitude, allowing computations to be performed on a workstation rather than a dedicated high-performance computer. In addition, we describe accuracy benefits of emulator-assisted DA when compared to simply using the emulator for forecasting (i.e., without DA). Our overall formulation is denoted AIAEDA (Artificial Intelligence Emulator Assisted Data Assimilation).

Automatic differentiation applied to excitations with projected entangled pair states

The excitation ansatz for tensor networks is a powerful tool for simulating the low-lying quasiparticle excitations above ground states of strongly correlated quantum many-body systems. Recently, the two-dimensional tensor network class of infinite projected entangled-pair states gained new ground state optimization methods based on automatic differentiation, which are at the same time highly accurate and simple to implement. Naturally, the question arises whether these new ideas can also be used to optimize the excitation ansatz, which has recently been implemented in two dimensions as well. In this paper, we describe a straightforward way to reimplement the framework for excitations using automatic differentiation, and demonstrate its performance for the Hubbard model at half filling.

PyCO2SYS v1.8: marine carbonate system calculations in Python

Oceanic dissolved inorganic carbon (TC) is the largest pool of carbon that substantially interacts with the atmosphere on human timescales. Oceanic TC is increasing through uptake of anthropogenic carbon dioxide (CO2), and seawater pH is decreasing as a consequence. Both the exchange of CO2 between the ocean and atmosphere and the pH response are governed by a set of parameters that interact through chemical equilibria, collectively known as the marine carbonate system. To investigate these processes, at least two of the marine carbonate system's parameters are typically measured – most commonly, two from TC, total alkalinity (AT), pH, and seawater CO2 fugacity (fCO2; or its partial pressure, pCO2, or its dry-air mole fraction, xCO2) – from which the remaining parameters can be calculated and the equilibrium state of seawater solved. Several software tools exist to carry out these calculations, but no fully functional and rigorously validated tool written in Python, a popular scientific programming language, was previously available. Here, we present PyCO2SYS, a Python package intended to fill this capability gap. We describe the elements of PyCO2SYS that have been inherited from the existing CO2SYS family of software and explain subsequent adjustments and improvements. For example, PyCO2SYS uses automatic differentiation to solve the marine carbonate system and calculate chemical buffer factors, ensuring that the effect of every modelled solute and reaction is accurately included in all its results. We validate PyCO2SYS with internal consistency tests and comparisons against other software, showing that PyCO2SYS produces results that are either virtually identical or different for known reasons, with the differences negligible for all practical purposes. We discuss insights that guided the development of PyCO2SYS: for example, the fact that the marine carbonate system cannot be unambiguously solved from certain pairs of parameters. Finally, we consider potential future developments to PyCO2SYS and discuss the outlook for this and other software for solving the marine carbonate system. The code for PyCO2SYS is distributed via GitHub (https://github.com/mvdh7/PyCO2SYS, last access: 23 December 2021) under the GNU General Public License v3, archived on Zenodo (Humphreys et al., 2021), and documented online (https://pyco2sys.readthedocs.io/en/latest/, last access: 23 December 2021).

Efficient gradient computation for optimization of hyperparameters

We are interested in learning the hyperparameters in a convex objective function in a supervised setting. The complex relationship between the input data to the convex problem and the desirable hyperparameters can be modeled by a neural network; the hyperparameters and the data then drive the convex minimization problem, whose solution is then compared to training labels. In our previous work [1], we evaluated a prototype of this learning strategy in an optimization-based sinogram smoothing plus FBP reconstruction framework. A question arising in this setting is how to efficiently compute (backpropagate) the gradient from the solution of the optimization problem, to the hyperparameters to enable end-to-end training. In this work, we first develop general formulas for gradient backpropagation for a subset of convex problems, namely the proximal mapping. To illustrate the value of the general formulas and to demonstrate how to use them, we consider the specific instance of 1-D quadratic smoothing (denoising) whose solution admits a dynamic programming (DP) algorithm. The general formulas lead to another DP algorithm for exact computation of the gradient of the hyperparameters. Our numerical studies demonstrate a 55%- 65% computation time savings by providing a custom gradient instead of relying on automatic differentiation in deep learning libraries. While our discussion focuses on 1-D quadratic smoothing, our initial results (not presented) support the statement that the general formulas and the computational strategy apply equally well to TV or Huber smoothing problems on simple graphs whose solutions can be computed exactly via DP.

A blind deconvolution algorithm based on backward automatic differentiation and its application to rolling bearing fault diagnosis

Blind deconvolution (BD) is an effective algorithm for enhancing the impulsive signature of rolling bearings. As a convex optimization problem, the existing BDs have poor optimization performance and cannot effectively enhance the impulsive signature excited by weak faults. Moreover, the existing BDs require manual derivation of the calculation process, which brings great inconvenience to the researcher's personalized design of the maximization criterion. A new BD algorithm based on backward automatic differentiation (BAD) is proposed, which is named BADBD. The calculation process does not require manual derivation so a general solution of BDs based on different maximization criteria is realized. BADBD constructs multiple cascaded filters to filter the raw vibration signal, which makes up for the deficiency of single filter performance. The filter coefficients are determined by Adam algorithm, which improves the optimization performance of the proposed BADBD. BADBD is compared with classic BDs by synthesized and real vibration signals. The results reveal superior capability of BADBD to enhance the impulsive signature and the fault diagnosis performance is significantly better than the classic BDs.

Automatic differentiation of thyroid scintigram by deep convolutional neural network: a dual center study

Background 99mTc-pertechnetate thyroid scintigraphy is a valid complementary avenue for evaluating thyroid disease in the clinic, the image feature of thyroid scintigram is relatively simple but the interpretation still has a moderate consistency among physicians. Thus, we aimed to develop an artificial intelligence (AI) system to automatically classify the four patterns of thyroid scintigram. Methods We collected 3087 thyroid scintigrams from center 1 to construct the training dataset (n = 2468) and internal validating dataset (n = 619), and another 302 cases from center 2 as external validating datasets. Four pre-trained neural networks that included ResNet50, DenseNet169, InceptionV3, and InceptionResNetV2 were implemented to construct AI models. The models were trained separately with transfer learning. We evaluated each model’s performance with metrics as following: accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), recall, precision, and F1-score. Results The overall accuracy of four pre-trained neural networks in classifying four common uptake patterns of thyroid scintigrams all exceeded 90%, and the InceptionV3 stands out from others. It reached the highest performance with an overall accuracy of 92.73% for internal validation and 87.75% for external validation, respectively. As for each category of thyroid scintigrams, the area under the receiver operator characteristic curve (AUC) was 0.986 for ‘diffusely increased,’ 0.997 for ‘diffusely decreased,’ 0.998 for ‘focal increased,’ and 0.945 for ‘heterogeneous uptake’ in internal validation, respectively. Accordingly, the corresponding performances also obtained an ideal result of 0.939, 1.000, 0.974, and 0.915 in external validation, respectively. Conclusions Deep convolutional neural network-based AI model represented considerable performance in the classification of thyroid scintigrams, which may help physicians improve the interpretation of thyroid scintigrams more consistently and efficiently.