Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training

In this paper, we introduce the Layer-Peeled Model, a nonconvex, yet analytically tractable, optimization program, in a quest to better understand deep neural networks that are trained for a sufficiently long time. As the name suggests, this model is derived by isolating the topmost layer from the remainder of the neural network, followed by imposing certain constraints separately on the two parts of the network. We demonstrate that the Layer-Peeled Model, albeit simple, inherits many characteristics of well-trained neural networks, thereby offering an effective tool for explaining and predicting common empirical patterns of deep-learning training. First, when working on class-balanced datasets, we prove that any solution to this model forms a simplex equiangular tight frame, which, in part, explains the recently discovered phenomenon of neural collapse [V. Papyan, X. Y. Han, D. L. Donoho, Proc. Natl. Acad. Sci. U.S.A. 117, 24652–24663 (2020)]. More importantly, when moving to the imbalanced case, our analysis of the Layer-Peeled Model reveals a hitherto-unknown phenomenon that we term Minority Collapse, which fundamentally limits the performance of deep-learning models on the minority classes. In addition, we use the Layer-Peeled Model to gain insights into how to mitigate Minority Collapse. Interestingly, this phenomenon is first predicted by the Layer-Peeled Model before being confirmed by our computational experiments.

Effect of Shot Peening on the Evolution of Scale on T91 Steel Exposed to Steam

Shot peening can be an effective solution for the prevention or retardation of scale formation, and subsequent exfoliation, upon exposure of the inner tube to steam in coal-fired power plants. In this study, specimens of T91 tubes were shot peened and then exposed to 1-bar steam for 100–1000 h at 650 °C, and were then analyzed using Vickers hardness test and microscopic techniques OM, SEM, TEM, etc. The analysis indicates that the oxide scales are typically Fe2O3 on the topmost layer, Fe3O4 below, and a FeCr2O4 spinel on the bottom in both shot peening treated and untreated specimens. However, the oxide scale thicknesses of shot peened specimens are thinner, indicating that shot peened specimens have better oxidation resistance. In addition, numerous defects, such as voids and micro-cracks, were found in the untreated specimens, which are believed to cause exfoliation of the uppermost Fe2O3 layers of the specimens exposed to steam for 800 and 1000 h. By contrast, the shot peened specimens maintained a dense contact oxide scale with fewer defects.

Crisscross multilayering of cell sheets

Simple hydrostatic skeletons such as the Hydra's consist of two stacked layers of cells perpendicularly oriented. Although this crisscross architecture can be recapitulated in vitro, little is known on the formation of such multilayers starting from a monolayer. In the present article, we show that bilayering of myoblasts results from the organization and activity of the cells originally in the monolayer which can be described as a contractile active nematic. As expected, most of the +1/2 topological defects that are associated with this nematic order self-propel. However, a subpopulation of these defects remains immobile. Perpendicular bilayering occurs exclusively at these motionless defects. Indeed, cells located at the head of these defects converge toward the (immobile) core and accumulate there until they start migrating on top of the tail of the first layer while the tail cells migrate in the opposite direction under the head cells. Since the cells keep their initial orientations, the two stacked layers end up perpendicularly oriented. This concerted process leading to a bilayer is dependent on the apical secretion of Extra Cellular Matrix (ECM) by the cells. Indeed, we evidence the presence of ECM between the cell layers and at the apical surface of the topmost layer. ECM molecules are oriented in the direction of the cells that produce them, which may guide the migration of the subsequent cell layers on their apical side.

Tuning the Surface Properties of Poly(Allylamine Hydrochloride)-Based Multilayer Films

The layer-by-layer (LbL) method of polyelectrolyte multilayer (PEM) fabrication is extremely versatile. It allows using a pair of any oppositely charged polyelectrolytes. Nevertheless, it may be difficult to ascribe a particular physicochemical property of the resulting PEM to a structural or chemical feature of a single component. A solution to this problem is based on the application of a polycation and a polyanion obtained by proper modification of the same parent polymer. Polyelectrolyte multilayers (PEMs) were prepared using the LbL technique from hydrophilic and amphiphilic derivatives of poly(allylamine hydrochloride) (PAH). PAH derivatives were obtained by the substitution of amine groups in PAH with sulfonate, ammonium, and hydrophobic groups. The PEMs were stable in 1 M NaCl and showed three different modes of thickness growth: exponential, mixed exponential-linear, and linear. Their surfaces ranged from very hydrophilic to hydrophobic. Root mean square (RMS) roughness was very variable and depended on the PEM composition, sample environment (dry, wet), and the polymer constituting the topmost layer. Atomic force microscopy (AFM) imaging of the surfaces showed very different morphologies of PEMs, including very smooth, porous, and structured PEMs with micellar aggregates. Thus, by proper choice of PAH derivatives, surfaces with different physicochemical features (growth type, thickness, charge, wettability, roughness, surface morphology) were obtained.

First Principle Research on Ga Rich GaAs0.5P0.5(001) β2(4×2) and As(P) rich β2(2×4) Reconstruction Surfaces With and Without Cs Adsorption

Based on first principle calculations, Ga rich and As(P) rich clean GaAs0.5P0.5(001) reconstruction surfaces and adsorbed surfaces with 0.125ML coverage of Cs at different sites are researched. Formation energy of Ga rich GaAs0.5P0.5(001) β2(4×2) reconstruction surface is smaller than that of As(P) rich one, and the work functions of Ga rich β 2 (4×2) and As(P) rich β2(2×4) surfaces are 4.657 eV and 5.187 eV, respectively. The adsorption energies of Cs adatoms on both surfaces are negative, showing that Cs adsorption is a stable exothermic process. The work functions of two surfaces both decrease after Cs adsorption, and the average variation of As(P) rich β2(2×4) surface is larger. Mulliken charge analysis shows that Cs adatoms transfer electrons to GaAsP substrate, resulting in Cs-GaAsP dioples which lower the work functions. When Cs atoms are located at D 2 of Ga rich surface and D 2 ' of As(P) rich surface, work function values of the two reconstruction surfaces reach the minimums, which are 2.834eV and 2.859eV, respectively. By calculating dipole moments, it can be found that Cs adatoms on the topmost layer form larger effective dipole moments with GaAsP substrate than the Cs atoms located in the trench.

Effect of the External Velocity on the Exfoliation Properties of Graphene from Amorphous SiO2 Surface

External action has a significant influence on the formation of high-quality graphene and the adhesion of graphene on the surface of the MEMS/NEMS device. The atomic-scale simulation and calculation can further study the exfoliation process of graphene by external actions. In multilayer graphene systems where graphene layers were simulated weakly contacted with SiO2 substrate, a constant vertical upward velocity (Vup) was applied to the topmost layer. Then two critical velocities were found, and three kinds of distinct exfoliation processes determined by critical upward velocities were observed in multilayer graphene systems. The first critical velocities are in the range of 0.5 Å/ps–3.18 Å/ps, and the second critical velocities are in the range of 9.5 Å/ps–12.1 Å/ps. When the Vup is less than the first critical velocity, all graphene layers will not be exfoliated. When Vup is between the first and second critical Vup, all layers can be exfoliated almost synchronously at last. When Vup is larger than the second critical Vup, the topmost layer can be exfoliated alone, transferring energy to the underlying layers, and the underlying layers are slowly exfoliated. The maximum exfoliation force to exfoliate the topmost layer of graphene is 3200 times larger than that of all graphene layers. Moreover, it is required 149.26 mJ/m2 to get monolayer graphene from multilayers, while peeling off all layers without effort. This study explains the difficulty to get monolayer graphene and why graphene falls off easily during the transfer process.

Atomic Layer Deposition of Insulating AlF3/Polyimide Nanolaminate Films

This article describes the deposition of AlF3/polyimide nanolaminate film by inorganic-organic atomic layer deposition (ALD) at 170 °C. AlCl3 and TiF4 were used as precursors for AlF3. Polyimide layers were deposited from PMDA (pyromellitic dianhydride, 1,2,3,5-benzenetetracarboxylic anhydride) and DAH (1,6-diaminohexane). With field-emission scanning electron microscopy (FESEM) and X-ray reflection (XRR) analysis, it was found that the topmost layer (nominally 10 nm in thickness) of the nanolaminate film (100 nm total thickness) changed when exposed to the atmosphere. After all, the effect on roughness was minimal. The length of a delay time between the AlF3 and polyimide depositions was found to affect the sharpness of the nanolaminate structure. Electrical properties of AlF3/polyimide nanolaminate films were measured, indicating an increase in dielectric constant compared to single AlF3 and a decrease in leakage current compared to polyimide films, respectively.

REE redistributions during granite weathering: Implications for Ce anomaly as a proxy for paleoredox states

Different responses of Ce to the redox state from those of the other light rare earth elements (LREEs) can be used to understand paleoredox states. To establish the possibility of using the Ce anomaly as a proxy for paleo-environments, we examined the mineralogical and chemical characteristics of bulk samples and REE-bearing minerals of a modern weathering profile developed on granite, by X-ray fluorescence analysis, laser-ablation inductively coupled plasma mass spectrometry, field emission electron microprobe analysis, field emission scanning electron microscopy, and X-ray diffractometry. Bulk samples showed no significant Ce-anomalies except for the topmost layer that had a positive Ce-anomaly reflecting significant loss of LREEs except for Ce. Allanite-(Ce), primary REE-bearing mineral, contributed to ~100% of La, Ce, Pr, and Nd in the parent rock, and gradually decreased in amount toward the topmost layer. Secondary cerianite-(Ce) [Ce(IV)O2] was observed in the weathering profile, especially at shallower depths. Secondary rhabdophane-(La), -(Ce), -(Nd), and -(Y) were also observed in the weathering profile but in less amounts in the topmost layer. The occurrences of rhabdophane-(La) and -(Nd) in contact with halloysite, a secondary clay mineral, suggest probable adsorption of REEs onto halloysite prior to their formation. Similar formation mechanisms are likely for rhabdophane-(Ce) that commonly occurred in grain boundaries and was usually formed in contact with halloysite. Rhabdophane-(Y) occurred in association with fluorapatite. The ratios of La, Pr, and Nd of rhabdophane-(La), -(Ce), and -(Nd) were similar to that of allanite-(Ce), suggesting that these LREEs are inherited from allanite-(Ce) and behave similarly before the formation of rhabdophane. Different negative Ce-anomaly values of rhabdophane [i.e., ~0.03–0.34 for rhabdophane-(La), -(Nd), and -(Y), and ~0.6 for rhabdophane-(Ce)] can result from a difference in intensity of the formation of cerianite-(Ce) prior to the precipitation of rhabdophane. We have classified LREE redistributions in both secondary minerals and bulk weathered samples during oxic weathering and suggested that Ce anomaly can provide useful information on anoxic weathering and thus atmospheric oxygen evolution in the Precambrian if Ce anomalies of both bulk samples and secondary REE-bearing minerals are determined.

Suppression of Hydrophobic Recovery in Photo-Initiated Chemical Vapor Deposition

Photo-initiated chemical vapor deposition (PICVD) functionalizes carbon nanotube (CNT)-enhanced porous substrates with a highly polar polymeric nanometric film, rendering them super-hydrophilic. Despite its ability to generate fully wettable surfaces at low temperatures and atmospheric pressure, PICVD coatings normally undergo hydrophobic recovery. This is a process by which a percentage of oxygenated functional group diffuse/re-arrange from the top layer of the deposited film towards the bulk of the substrate, taking the induced hydrophilic property of the material with them. Thus, hydrophilicity decreases over time. To address this, a vertical chemical gradient (VCG) can be deposited onto the CNT-substrate. The VCG consists of a first, thicker highly cross-linked layer followed by a second, thinner highly functionalized layer. In this article, we show, through water contact angle and XPS measurements, that the increased cross-linking density of the first layer can reduce the mobility of polar functional groups, forcing them to remain at the topmost layer of the PICVD coating and to suppress hydrophobic recovery. We show that employing a bi-layer VCG suppresses hydrophobic recovery for five days and reduces its effect afterwards (contact angle stabilizes to 42 ± 1° instead of 125 ± 3°).