Accurate and Precise Prediction of Soil Properties from a Large Mid-Infrared Spectral Library

Diffuse reflectance spectroscopy (DRS) is emerging as a rapid and cost-effective alternative to routine laboratory analysis for many soil properties. However, it has primarily been applied in project-specific contexts. Here, we provide an assessment of DRS spectroscopy at the scale of the continental United States by utilizing the large (n > 50,000) USDA National Soil Survey Center mid-infrared spectral library and associated soil characterization database. We tested and optimized several advanced statistical approaches for providing routine predictions of numerous soil properties relevant to studying carbon cycling. On independent validation sets, the machine learning algorithms Cubist and memory-based learner (MBL) both outperformed random forest (RF) and partial least squares regressions (PLSR) and produced excellent overall models with a mean R2 of 0.92 (mean ratio of performance to deviation = 6.5) across all 10 soil properties. We found that the use of root-mean-square error (RMSE) was misleading for understanding the actual uncertainty about any particular prediction; therefore, we developed routines to assess the prediction uncertainty for all models except Cubist. The MBL models produced much more precise predictions compared with global PLSR and RF. Finally, we present several techniques that can be used to flag predictions of new samples that may not be reliable because their spectra fall outside of the calibration set.

Download Full-text

Is Standardization Necessary for Sharing of a Large Mid-Infrared Soil Spectral Library?

Sensors ◽

10.3390/s20236729 ◽

2020 ◽

Vol 20 (23) ◽

pp. 6729

Author(s):

Shree R. S. Dangal ◽

Jonathan Sanderman

Keyword(s):

Soil Properties ◽

Predictive Models ◽

Soil Samples ◽

Soil Survey ◽

Least Squares Regression ◽

Spectral Library ◽

Calibration Transfer ◽

Mid Infrared ◽

Calibration Models ◽

Spectral Libraries

Recent developments in diffuse reflectance soil spectroscopy have increasingly focused on building and using large soil spectral libraries with the purpose of supporting many activities relevant to monitoring, mapping and managing soil resources. A potential limitation of using a mid-infrared (MIR) spectral library developed by another laboratory is the need to account for inherent differences in the signal strength at each wavelength associated with different instrumental and environmental conditions. Here we apply predictive models built using the USDA National Soil Survey Center–Kellogg Soil Survey Laboratory (NSSC-KSSL) MIR spectral library (n = 56,155) to samples sets of European and US origin scanned on a secondary spectrometer to assess the need for calibration transfer using a piecewise direct standardization (PDS) approach in transforming spectra before predicting carbon cycle relevant soil properties (bulk density, CaCO3, organic carbon, clay and pH). The European soil samples were from the land use/cover area frame statistical survey (LUCAS) database available through the European Soil Data Center (ESDAC), while the US soil samples were from the National Ecological Observatory Network (NEON). Additionally, the performance of the predictive models on PDS transfer spectra was tested against the direct calibration models built using samples scanned on the secondary spectrometer. On independent test sets of European and US origin, PDS improved predictions for most but not all soil properties with memory based learning (MBL) models generally outperforming partial least squares regression and Cubist models. Our study suggests that while good-to-excellent results can be obtained without calibration transfer, for most of the cases presented in this study, PDS was necessary for unbiased predictions. The MBL models also outperformed the direct calibration models for most of the soil properties. For laboratories building new spectroscopy capacity utilizing existing spectral libraries, it appears necessary to develop calibration transfer using PDS or other calibration transfer techniques to obtain the least biased and most precise predictions of different soil properties.

Download Full-text

Can Agricultural Management Induced Changes in Soil Organic Carbon Be Detected Using Mid-Infrared Spectroscopy?

Remote Sensing ◽

10.3390/rs13122265 ◽

2021 ◽

Vol 13 (12) ◽

pp. 2265

Author(s):

Jonathan Sanderman ◽

Kathleen Savage ◽

Shree Dangal ◽

Gabriel Duran ◽

Charlotte Rivard ◽

...

Keyword(s):

Organic Carbon ◽

Soil Organic Carbon ◽

Soil Carbon ◽

Diffuse Reflectance Spectroscopy ◽

Low Cost ◽

Soil Survey ◽

Mid Infrared ◽

Mir Spectroscopy ◽

Induced Changes ◽

Carbon Change

A major limitation to building credible soil carbon sequestration programs is the cost of measuring soil carbon change. Diffuse reflectance spectroscopy (DRS) is considered a viable low-cost alternative to traditional laboratory analysis of soil organic carbon (SOC). While numerous studies have shown that DRS can produce accurate and precise estimates of SOC across landscapes, whether DRS can detect subtle management induced changes in SOC at a given site has not been resolved. Here, we leverage archived soil samples from seven long-term research trials in the U.S. to test this question using mid infrared (MIR) spectroscopy coupled with the USDA-NRCS Kellogg Soil Survey Laboratory MIR spectral library. Overall, MIR-based estimates of SOC%, with samples scanned on a secondary instrument, were excellent with the root mean square error ranging from 0.10 to 0.33% across the seven sites. In all but two instances, the same statistically significant (p < 0.10) management effect was found using both the lab-based SOC% and MIR estimated SOC% data. Despite some additional uncertainty, primarily in the form of bias, these results suggest that large existing MIR spectral libraries can be operationalized in other laboratories for successful carbon monitoring.

Download Full-text

Determination of Wavelength Accuracy in the Near-Infrared Spectral Region Based on NIST's Infrared Transmission Wavelength Standard SRM 1921

Applied Spectroscopy ◽

10.1366/0003702001949573 ◽

2000 ◽

Vol 54 (3) ◽

pp. 450-455 ◽

Cited By ~ 5

Author(s):

Stephen R. Lowry ◽

Jim Hyatt ◽

William J. McCarthy

Keyword(s):

Reference Material ◽

Standard Reference Material ◽

Spectral Region ◽

Near Infrared ◽

Cost Effective ◽

Infrared Transmission ◽

Internal Reference ◽

Mid Infrared ◽

Infrared Spectral Region ◽

Infrared Spectral

A major concern with the use of near-infrared (NIR) spectroscopy in many QA/QC laboratories is the need for a simple reliable method of verifying the wavelength accuracy of the instrument. This requirement is particularly important in near-infrared spectroscopy because of the heavy reliance on sophisticated statistical vector analysis techniques to extract the desired information from the spectra. These techniques require precise alignment of the data points between the vectors corresponding to the standard and sample spectra. The National Institute of Standards and Technology (NIST) offers a Standard Reference Material (SRM 1921) for the verification and calibration of mid-infrared spectrometers in the transmittance mode. This standard consists of a 38 μm-thick film of polystyrene plastic. While SRM 1921 works well as a mid-infrared standard, a thicker sample is required for use as a routine standard in the near-infrared spectral region. The general acceptance and proven reliability of polystyrene as a standard reference material make it a very good candidate for a cost-effective NIR standard that could be offered as an internal reference for every instrument. In this paper we discuss a number of the parameters in a Fourier transform (FT)-NIR instrument that can affect wavelength accuracy. We also report a number of experiments designed to determine the effects of resolution, sample position, and optics on the wavelength accuracy of the system. In almost all cases the spectral reproducibility was better than one wavenumber of the values extrapolated from the NIST reference material. This finding suggests that a thicker sample of polystyrene plastic that has been validated with the SRM 1921 standard would make a cost-effective reference material for verifying wavelength accuracy in a medium-resolution FT-NIR spectrometer.

Download Full-text

Estimating Soil Properties and Nutrients by Visible and Infrared Diffuse Reflectance Spectroscopy to Characterize Vineyards

Agronomy ◽

10.3390/agronomy11101895 ◽

2021 ◽

Vol 11 (10) ◽

pp. 1895

Author(s):

José Ramón Rodríguez-Pérez ◽

Víctor Marcelo ◽

Dimas Pereira-Obaya ◽

Marta García-Fernández ◽

Enoc Sanz-Ablanedo

Keyword(s):

Soil Properties ◽

Diffuse Reflectance Spectroscopy ◽

Full Range ◽

Cost Effective ◽

Reflectance Spectroscopy ◽

Least Squares Regression ◽

Standard Normal Variate ◽

Standard Normal ◽

Western Spain ◽

Shortwave Infrared

Visible, near, and shortwave infrared (VIS-NIR-SWIR) reflectance spectroscopy, a cost-effective and rapid means of characterizing soils, was used to predict soil sample properties for four vineyards (central and north-western Spain). Sieved and air-dried samples were measured using a portable spectroradiometer (350–2500 nm) and compared for pistol grip (PG) versus contact probe (CP) setups. Raw data processed using standard normal variate (SVN) and detrending transformation (DT) were grouped into four subsets (VIS: 350–700 nm; NIR: 701–1000 nm; SWIR: 1001–2500 nm; and full range: 350–2500 nm) in order to identify the most suitable range for determining soil characteristics. The performance of partial least squares regression (PLSR) models in predicting soil properties from reflectance spectra was evaluated by cross-validation. The four spectral subsets and transformed reflectances for each setup were used as PLSR predictor variables. The best performing PLSR models were obtained for pH, electrical conductivity, and phosphorous (R2 values above 0.92), while models for sand, nitrogen, and potassium showed moderately good performances (R2 values between 0.69 and 0.77). The SWIR subset and SVN + DT processing yielded the best PLSR models for both the PG and CP setups. VIS-NIR-SWIR reflectance spectroscopy shows promise as a technique for characterizing vineyard soils for precision viticulture purposes. Further studies will be carried out to corroborate our findings.

Download Full-text

Visible, near infrared, mid infrared or combined diffuse reflectance spectroscopy for simultaneous assessment of various soil properties

Geoderma ◽

10.1016/j.geoderma.2005.03.007 ◽

2006 ◽

Vol 131 (1-2) ◽

pp. 59-75 ◽

Cited By ~ 1036

Author(s):

R.A. Viscarra Rossel ◽

D.J.J. Walvoort ◽

A.B. McBratney ◽

L.J. Janik ◽

J.O. Skjemstad

Keyword(s):

Soil Properties ◽

Diffuse Reflectance ◽

Near Infrared ◽

Diffuse Reflectance Spectroscopy ◽

Reflectance Spectroscopy ◽

Mid Infrared ◽

Simultaneous Assessment

Download Full-text

Prediction of Soil Fertility Properties from a Globally Distributed Soil Mid-Infrared Spectral Library

Soil Science Society of America Journal ◽

10.2136/sssaj2009.0218 ◽

2010 ◽

Vol 74 (5) ◽

pp. 1792-1799 ◽

Cited By ~ 86

Author(s):

Thomas Terhoeven-Urselmans ◽

Tor-Gunnar Vagen ◽

Otto Spaargaren ◽

Keith D. Shepherd

Keyword(s):

Soil Fertility ◽

Spectral Library ◽

Mid Infrared ◽

Infrared Spectral ◽

Globally Distributed

Download Full-text

Searching a mid-infrared spectral library of solids and liquids with spectra of mixtures

Vibrational Spectroscopy ◽

10.1016/s0924-2031(96)00067-7 ◽

1997 ◽

Vol 14 (1) ◽

pp. 9-17 ◽

Cited By ~ 9

Author(s):

Chi-Shi Chen ◽

Yue Li ◽

Chris W Brown

Keyword(s):

Spectral Library ◽

Mid Infrared ◽

Infrared Spectral

Download Full-text

Evaluating near infrared spectroscopy for field prediction of soil properties

Soil Research ◽

10.1071/sr09005 ◽

2009 ◽

Vol 47 (7) ◽

pp. 664 ◽

Cited By ~ 25

Author(s):

Budiman Minasny ◽

Alex B. McBratney ◽

Leo Pichon ◽

Wei Sun ◽

Michael G. Short

Keyword(s):

Soil Properties ◽

Near Infrared ◽

Agricultural Land ◽

Diffuse Reflectance Spectroscopy ◽

Soil Samples ◽

Soil Mapping ◽

Digital Soil Mapping ◽

Soil Conditions ◽

Spectral Library ◽

Field Samples

This paper demonstrates the application of near infrared diffuse reflectance spectroscopy (NIR-DRS) measurements as part of digital soil mapping. We also investigate whether calibration functions developed from a spectral library can be used for rapid characterisation of soil properties in the field. Soil samples were collected along 24 toposequences in the Pokolbin irrigation district, ~7 km2 of predominantly agricultural land in the Hunter Valley, NSW, Australia. Soil samples at 2 depths: 0–0.10 and 0.40–0.50 m were collected. The soil samples were scanned using NIR under 3 different conditions: field condition, dried unground, and dried ground. A separate spectral library containing soil laboratory measurements was used to develop functions to predict 3 main soil properties from NIR spectra (total C content, clay content, and sum of exchangeable cations). The absorbance spectra were found to be different for the 3 soil conditions. The field spectra appear to have higher absorbance, followed by dried unground samples and then dried ground samples. Although most spectral signatures or peaks were similar for the 3 soil conditions, field samples appear to have higher absorbance, particularly at 1400 nm and 1900 nm. The convex hull of the first 2 principal components of the soil spectra is an easy tool to evaluate the similarity of spectra from a calibration set to an observation. For field prediction, samples need to be calibrated using field samples. Finally, this study shows that NIR-DRS measurement is a useful part of digital soil mapping.

Download Full-text