How Can We Know What Language Models Know?

Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .

An international comparison of the second derivative of COVID-19 deaths after implementation of social distancing measures

This work compares deaths for confirmed COVID-19 cases in China to eight other countries, Italy, Spain, France, USA, UK, Germany, Netherlands and South Korea. After implementing varying intensities and timing of social distancing measures, several appear to be converging onto the decline in the daily growth rate of deaths, or relative second derivative of total deaths, seen in China after the implementation an aggressive social distancing policy. By calculating future trajectories in these countries based on the observed Chinese fatality statistics, an estimate of the total deaths and maximum daily death rates over a defined period of time is made. Our lower bound estimate for the United Kingdom based on the real data approximates the lower bound estimate of the recent modelling study of Ferguson et al. [1]. These results suggest there may be a threshold of effective public health intervention. Our method of viewing the data may be helpful in monitoring the course of the epidemic, judging the effectiveness of implementation, and monitoring the relaxation of social distancing.

Diaphragm axial capacity for external diaphragm connections (EDCs) in square CFST column structures

This paper evaluates external diaphragm axial capacity in moment frame structures with square concrete-filled steel tubular (CFST) columns considering bidirectional loading. Three design methods were considered: (1) the CIDECT method; (2) the equivalent beam method; and (3) the tie method. Finite element analyses were conducted to investigate the behaviour of an external diaphragm plate connected to a square CFST column under varied bidirectional diaphragm axial forces. It is shown that the perpendicular diaphragm axial forces did not reduce the diaphragm axial capacity significantly, which is consistent with the assumptions made by the CIDECT method and the tie method. The CIDECT method, in some cases, was not conservative. Among the considered methods, the tie method was the most justifiable method, although in some cases the capacity predictions were too conservative. The tie method was later modified by considering the contribution of the steel tube in addition to the diaphragm plate in calculating the diaphragm axial capacity. The modified tie method was shown to accurately predict a lower bound estimate of the capacity of an external diaphragm connection.

Convective Heat Transfer in CANDU Spent Fuel Racks After a Loss of Coolant

The event at the Fukushima Daiichi Spent Fuel Pools (SFPs) has renewed interest in quantifying the safety margins related to loss of coolant accidents in Irradiated Fuel Bays (IFBs). Thermal-hydraulic analyses of exposed spent CANDU fuel has been limited to a small number of bundles due to its complex bundle geometry and open rack design. This paper presents a process to predict the steady state temperature and velocity of air as it passes through a rack of spent fuel using analytical models and Computational Fluid Dynamics (CFDs) techniques. The scenario acts as lower bound estimate for the effectiveness of convection during a complete loss of coolant in a fuel bay by examining the heat-up of a stand-alone rack without flow resistance of the bundles. The correct incorporation of flow resistance is a necessary step before conclusions are made about the available safety margins of irradiated fuel bays.