Miniaturised visible and near-infrared spectrometers for assessing soil health indicators in mine site rehabilitation

Mining can cause severe disturbances to the soil, which underpins the viability of terrestrial ecosystems. Post-mining rehabilitation relies on measuring soil properties that are critical indicators of soil health. Soil visible–near-infrared (vis–NIR) spectroscopy is rapid, accurate, and cost-effective for estimating a range of soil properties. Recent advances in infrared detectors and microelectromechanical systems (MEMSs) have produced miniaturised, relatively inexpensive spectrometers. Here, we evaluate the spectra from four miniaturised visible and NIR spectrometers, some combinations, and a full-range vis–NIR spectrometer for modelling 29 soil physical, chemical, and biological properties used to assess soil health at mine sites. We collected topsoil samples from reference, undisturbed native vegetation, and stockpiles from seven mines in Western Australia. We evaluated the spectrometers' repeatability and the accuracy of spectroscopic models built with seven statistical and machine learning algorithms. The spectra from the visible spectrometer could estimate sand, silt, and clay with similar or better accuracy than the NIR spectrometers. However, the spectra from the NIR spectrometers produced better estimates of soil chemical and biological properties. By combining the miniaturised visible and NIR spectrometers, we improved the accuracy of their soil property estimates, which were similar to those from the full-range spectrometer. The miniaturised spectrometers and combinations predicted 24 of the 29 soil properties with moderate or greater accuracy (Lin's concordance correlation, ρ_c≥0.65). The repeatability of the NIR spectrometers was similar to that of the full-range, portable spectrometer. The miniaturised NIR spectrometers produced comparably accurate soil property estimates to the full-range portable system which is an order of magnitude more expensive, particularly when combined with the visible range sensor. Thus, the miniaturised spectrometers could form the basis for a rapid, cost-effective soil diagnostic capacity to support mine site rehabilitation and deliver significant positive economic and environmental outcomes.

1 Introduction

Soil health underpins the viability of terrestrial ecosystems, whether natural or human-altered (e.g. agricultural, post-mining) and provides a variety of functions essential for life on Earth (Jeffrey, 2017; Timmis and Ramos, 2021). Healthy soil supports above- and below-ground biodiversity, plant growth, agricultural productivity, and a suite of ecological functions and ecosystem services (Timmis and Ramos, 2021). Consequently, returning functional soils to a site following disturbance is critical to achieving sustainable and resilient rehabilitation (e.g. reinstating a level of ecosystem productivity or functioning) or ecological restoration (e.g. assisting the recovery of an ecosystem that has been degraded, damaged, or destroyed; Gann et al., 2019). Some of the most severe disturbances to soil result from surface mining (Cooke and Johnson, 2002; Cross et al., 2017). Rehabilitation or ecological restoration is often a regulatory requirement for mining companies to undertake during mine closure (Manero et al., 2021). There is an increasing expectation that mining companies return functional, resilient, and biodiverse native ecosystems to lands where mining has occurred. However, rehabilitation or ecological restoration can be challenging on mined lands because the substrates generated by mining, such as tailings (fine particulate materials), waste rock, and salvaged topsoil (e.g. Stock et al., 2020), can be different from undisturbed homologues (Munoz-Rojas et al., 2016; Cross and Lambers, 2021; Cross et al., 2018). Different approaches to rehabilitation and ecological restoration practices such as landform design and contouring, topsoil return, seeding, and planting can significantly influence soil characteristics and other ecosystem attributes. Ensuring mined lands are placed on favourable ecological trajectories requires a fundamental understanding of the edaphic conditions of the pre-disturbance landscape and mined materials and how these conditions might influence soil properties and the capacity of soil to support ecological functioning.

Soil health, which we define as the capacity of soil to sustain biodiversity, productivity, and ecological functioning and to provide ecosystem services, represents an intricate series of interactions between important soil physical, chemical, and biological properties (Lehmann et al., 2020). Soil physical properties are essential for providing air, water, gaseous exchanges, and habitat; chemical properties for moderating soil reactions and nutrient transformations and availability; and biological properties for nutrient cycling (Lal, 2004). Soil health is fundamental to plant productivity and landscape stability (Turner et al., 2018). It is also essential to soil functioning (e.g. a medium for plant growth, habitat for soil organisms, carbon storage), which underpins most post-mining land uses, such as conservation and the reinstatement of native ecosystems to alternative uses such as agriculture. Assessing soil health by examining key physical, chemical, and biological indicators can help to guide, monitor, and evaluate ecological trajectories following restoration or rehabilitation (Rinot et al., 2019). Failure to understand and effectively manage soil health will likely result in undesirable trajectories with adverse and often cascading, long-term harmful impacts on biodiversity, ecosystem productivity, and resilience.

Assessment of soil health in post-mining rehabilitation and ecological restoration, when undertaken, remains typically an analysis of key indicators (e.g. soil nutrient concentrations, pH, electrical conductivity, cation exchange capacity) on composite samples collected from representative locations around a site. Assessment is generally undertaken periodically from the beginning of rehabilitation or ecological restoration activities to monitor changes in indicators over time, with collected soil samples analysed in a laboratory to provide average values of the soil properties. However, conventional assessment is time-consuming and expensive because laboratory analyses require elaborate methods and specialised equipment and procedures and can be prone to errors resulting from inappropriate sampling, transportation, preparation, or analysis (Viscarra Rossel and Bouma, 2016). These constraints often result in the collection of only a few samples, limiting spatial and temporal representation and the ability of sampling to characterise soil variability at a site adequately. Additionally, practitioners must wait for laboratory results to be returned following sample submission, limiting their ability to adapt management and intervention activities rapidly. Therefore, there is a need for a scientifically robust diagnostic capacity to rapidly, accurately, and inexpensively measure key soil properties that are indicators of soil health in the field.

Many soil properties that are considered key indicators of soil health, such as organic carbon, texture, water content, cation exchange capacity, pH, microbial biomass, and diversity, can be modelled with visible–near-infrared spectra (vis–NIR; 400–2500 nm) (e.g. Viscarra Rossel et al., 2006; Stenberg et al., 2010; Guerrero et al., 2010; Soriano-Disla et al., 2014; Yang et al., 2019). The modelling is possible because soil properties can be multivariately related to the wavelengths in the spectra, which contain information on the inherent composition of the soil, which comprises minerals, organic compounds, and water (Viscarra Rossel and Behrens, 2010). Thus, these spectra can describe soil qualitatively and quantitatively (Nocita et al., 2015). Broad, weak absorptions at wavelengths smaller than 1000 nm can result from chromophores and iron oxides; narrow, well-defined absorptions at wavelengths between 1400–1900 nm are due to hydroxyl bonds and water; absorptions at wavelengths around 2200 nm occur from clay minerals; and organic matter absorbs in different regions throughout the visible and NIR range. Vis–NIR spectroscopy also provides information on soil particle size and thus information on the soil matrix (Stenberg et al., 2010). Hence, in addition to soil properties, soil type and soil horizons can also be determined using vis–NIR spectra (Viscarra Rossel and Webster, 2011).

There are advantages to using the spectroscopic method. First, spectroscopic measurements are highly reproducible (Stenberg et al., 2010). Once spectroscopic models of soil properties are derived and validated, one can use them to estimate the values of those properties where those measurements are lacking and would be too expensive to make using conventional laboratory methods (Viscarra Rossel et al., 2006; Nocita et al., 2015). Spectroscopic models can be built with multivariate regressions, such as partial least-squares regression (PLSR) or machine learning methods such as support vector machines (SVMs), regression trees, or neural networks (Viscarra Rossel and Behrens, 2010) or, more recently, deep learning (e.g. Shen and Viscarra Rossel, 2021). Second, large databases of soil spectra are being developed to help meet the growing demand for soil information to evaluate and monitor soil at a range of scales (Viscarra Rossel and Webster, 2012; Orgiazzi et al., 2018; Viscarra Rossel et al., 2016; Shen et al., 2022). Third, as technologies develop, spectrometers have become cheaper, smaller, more portable, and accessible. Emerging infrared detector technologies are being used to produce miniaturised hand-held instruments that are rugged and affordable, also using microelectromechanical structures (MEMSs) (Christian and Ford, 2021; Johnson, 2015), thin-film filters, light-emitting diodes (LEDs), fibre optics, and high-performance detector arrays (Coates, 2014). As a result, there is growing interest in using miniaturised spectrometers for characterising soil properties (Tang et al., 2020; Ng et al., 2020; Sharififar et al., 2019). They provide a unique opportunity to develop a portable and cost-efficient soil health diagnostic capacity for mine site rehabilitation. However, these spectrometers have narrower spectral ranges and lower resolutions than portable or research-grade spectrometers, which are more commonly used to model soil properties (Stenberg et al., 2010; Soriano-Disla et al., 2014). A narrower spectral range and reduced spectral resolution might detriment a miniaturised spectrometer's capacity for estimating soil properties and developing a cost-effective soil health diagnostic solution. Therefore, we must thoroughly assess these spectrometers, the repeatability of their measurements, and their capacity to accurately estimate a wide range of soil physical, chemical, and biological properties deemed to be indicators of soil health. We have not found any published reports on such assessments in the literature.

Thus the objective of our study was to assess various commercially available miniaturised visible and NIR spectrometers (both independently and in combination) and one portable research-grade vis–NIR spectrometer for the capacity to estimate a wide range of soil chemical, physical, and biological properties typically used to assess soil health in Australian mine site soil rehabilitation. Specifically, our aims were to do the following:

• evaluate the repeatability of the spectroscopic measurements with each spectrometer;

• evaluate the modelling of 29 soil physical, chemical, and biological properties using the spectra from each spectrometer and combinations with seven statistical and machine learning algorithms;

• quantify the accuracy of the model estimates and the effect of repeatability on the estimates; and

• derive an overall score for each spectrometer and combinations, which accounts for their accuracy and repeatability.

The many soil properties and spectrometers and the assessment of repeatability and accuracy from modelling with various algorithms provide a comprehensive evaluation of spectroscopy for cost-effective mine site soil rehabilitation.

2 Methods

Seven mine sites in Western Australia were selected as study sites, representing a range of climatic conditions, soil types, vegetation assemblages, and commodity types (Table 1, Fig. 1).

Table 1 Resource type mined, predominant soil type, dominant vegetation type, and climate type for each of the mine sites included in the study. Note that Mine F has requested non-disclosure of resource mined.

Sites included a mineral sands mine (Fig. 1a), two iron ore mines (Fig. 1b, c), a copper–cobalt–nickel mine (Fig. 1d), a nickel mine (Fig. 1e), a bauxite mine (Fig. 1g), and a mine for which the commodity will remain undisclosed (anonymity requested; Fig. 1f). The soil types are orders from the Australian Soil Classification (Isbell, 2002).

Figure 1 Location of mine sites within Western Australia from which soil samples were collected for spectroscopic analysis. Soils were collected from a variety of vegetation types in different climatic regions, including from (a) a mineral sands mine within banksia woodland in a warm-summer Mediterranean region, (b, c) two iron ore mines within savanna woodland in the hot desert region, (d) a copper–cobalt–nickel mine within mallee woodland in the warm semi-arid region, (e) a nickel mine within open shrubland and grassland in the hot desert region, and (f, g) a bauxite mine and another undisclosed commodity mine both within jarrah forest in the warm-summer Mediterranean region. Inset: location of Western Australia within Australia.

2.1 Sampling design

We designed the sampling effort to cover a broad range of mining contexts. We sampled top-layer stockpiled and undisturbed reference soil samples from seven mines with differing soil types, climates, vegetation assemblages, and commodity. At each of the seven mine sites, three plots were established in undisturbed native vegetation (hereafter referred to as “reference plots”) and four in topsoil stockpiles (i.e. salvaged natural topsoil stockpiled for later re-spreading) of varying age (ages range from 3 months to 29 years) in summer 2018/2019. Five 5 m × 5 m subplots were established at each reference and stockpile plot, with five replicate soil samples collected at random from the top 0–20 cm soil layer in each subplot using a 5 cm diameter soil auger. At the youngest stockpile at each mine, five additional samples were taken from the 50–70 cm depth, which correspond roughly to the top layer of the original soil before stockpiling. The five soil samples were bulked and homogenised to produce one composite sample per subplot. Sterile nitrile gloves were worn when collecting soils, sampling equipment was sterilised with a bleach solution between all samples, and gloves were changed between plots to prevent cross-contamination. Subsamples were taken from each composite sample and stored at −20 ^∘C until DNA could be extracted (see below), while the remainder of the composites were dried in an oven at 40 ^∘C for 48 h before being sieved (2 mm gauge). Subsamples of the dried, composite soil were reserved for chemical analysis, assessment of carbon dioxide production, and analyses of soil microbial community composition. In total, 280 soil samples were collected from the seven mine sites. (i.e. At each mine there were three reference plots, four stockpile plots, and one young stockpile plot. Each plot consisted of five subplots, and one composite sample was collected from each subplot.)

2.2 Conventional soil analyses

To provide comparison data for the 29 soil chemical, physical, and biological soil properties to be assessed using spectroscopic methods, soil organic carbon content (Walkley and Black, 1934), potassium and phosphorus content (Colwell, 1965), pH (in a 1:5 soil to 0.01 M CaCl₂ and H₂O), electrical conductivity, sulfur, ammonium nitrogen, nitrate nitrogen, boron, trace elements (DTPA; copper, zinc, manganese, iron), and exchangeable cations (calcium, magnesium, sodium, potassium, aluminium) were determined for all composite samples using analytical chemistry methods sensu (Rayment and Lyons, 2010). Soil particle size (percentage sand, silt, clay) and bulk density (BD) were also assessed for each soil sample (Rai et al., 2017). Microbial activity of each soil sample was assessed using the Solvita 1-day CO₂ test (Haney et al., 2008; Munoz-Rojas et al., 2016). The test was conducted as per the manufacturer’s instructions (2019 SOP; Woods End Laboratories Inc., 2018–2019); briefly, 30 cm³ of dried soil was re-wet with 9 mL of water and placed in a sealed container with a Solvita CO₂ probe for 24 h, after which a digital colour reader (for use with the CO₂ test; Solvita, Woods End Laboratories, USA) was used to measure the volume of carbon dioxide produced.

Soil microbial community composition was determined by extracting DNA from 250 mg of homogenised soil samples (DNeasy PowerSoil HTP 96 Kit; Qiagen, Germany). DNA quality and level of inhibition were checked through quantitative PCR (qPCR), with negative PCR controls included (Murray et al., 2015). Primer sets used targeted the V4 16S rRNA region for bacteria (Turner et al., 1999; Caporaso et al., 2011) and the Internal Transcribed Spacer 2 for fungi (Ihrmark et al., 2012; White et al., 1990). Single fusion sequencing (see the Supplement “DNA Sequencing and Bioinformatics Methodology”) was performed at the Trace and Environmental DNA laboratory, Curtin University (Bentley, WA) on the MiSeq platform (Illumina, USA) as per van der Heyde et al. (2021). Sequences were de-multiplexed and quality-filtered, and error rates were estimated and de-replicated to leave amplicon sequence variants. Taxonomy was assigned based on reference databases (SILVA (Quast et al., 2013) for bacteria and UNITE8.2 (Nilsson et al., 2019) for fungi). Alpha diversity and richness of fungal and bacterial taxa were calculated from sequence data using the Shannon index (Wagner et al., 2018).

The statistical distributions of electrical conductivity, organic C, total N, K (Colwell), B, S, Cu, Fe, Mn, exchangeable Mg and Na, ammonium nitrogen, nitrate nitrogen, and CO₂ exhibited strong positive skews and were thus transformed using logarithms to approximate normal distributions prior to use in spectroscopic modelling.

2.3 Soil spectroscopy

For spectroscopic analysis, the ≤2 mm sieved composite subsamples were each mixed and then placed in a Petri dish. Diffuse reflectance spectra were recorded using five spectrometers with different spectral ranges, resolutions, dimensions, and costs (Table 2). We measured the soils following protocols described in Viscarra Rossel et al. (2016) (Appendix B). The spectrometers were switched on an hour before measurements, and the control software of each instrument was set up to record (and average) 30 readings per soil sample measurement and 50 readings per calibration measurement. Calibration was performed with a Halon white reference (Spectralon^®) and dark internal reference. The spectrometers were recalibrated every 10 measures. We performed the spectroscopic measurements of the soil samples in two separate rounds (i.e. in replicate), by the same analyst and under the same laboratory conditions.

Table 2 Spectral range, resolution, price, weight, and dimensions of miniaturised and portable spectrometers used in this study.

The reflectance (R) spectra of the soil samples recorded with each instrument were transformed to apparent absorbance using log${}_{\mathrm{10}}(\mathrm{1}/R)$ and interpolated to 10 nm intervals to attain a consistent wavelength interval. Since some of the miniaturised spectrometers measured narrow and specific spectral ranges, we combined spectrometers A_350−830 with B_1750−2150, A_350−830 with C_1450−2450, and A_350−830 with D_1300−2600 to cover more of the vis–NIR range.

2.4 Spectroscopic modelling

Given the large number of soil properties, spectrometers, and algorithms and the assessment of both spectrometer repeatability and model accuracy, the experiments became extremely complex and computationally intensive. To improve the computational efficiency of the study, we aggregated the data from the 280 subplots into 56 plots by averaging the spectra and soil properties (Fig. 2). Subplot samples were similar, and we assumed that aggregating them would not seriously affect the variability in the data and the modelling. Conclusions drawn from the results of the 56 plot data were compared to results from the 280 subplot data but using only the best spectrometers and algorithm (see below, Sect. 2.4.4 “Assessment on data from subplots”).

Figure 2 Study design for data collection and spectroscopic modelling. Data collection describes the measurement of soil properties and collection of spectra. Assessment on plots shows the spectroscopic modelling for the assessment of algorithms and spectrometers. Assessment on subplots validates the modelling with the best algorithm and spectrometers on subplot data.

2.4.1 Assessment of algorithms with data from plots

Multivariate modelling is fundamental for assessing the spectrometers. This type of modelling, and particularly machine learning, is largely dependent on the data set used, and there is not a single “best” method for all applications. Using a single algorithm could lead to inaccurate conclusions. To prevent both over- or understating the capability of these spectrometers, we used seven statistical and machine learning algorithms that have been reported for soil spectroscopic modelling (Viscarra Rossel and Behrens, 2010; Liu et al., 2016; Yang et al., 2022; Song et al., 2021). These were partial least-squares regression (PLSR) (Wold et al., 2001), random forest (RF) (Breiman, 2001), support vector machines (SVMs) (Vapnik, 1999), Cubist (Quinlan, 1992), extreme gradient boosting (XGBoost) (Chen et al., 2015), and Gaussian process regression with linear (GPRL) and polynomial (GPRP) kernels (Rasmussen, 2003). They account for linear responses (PLSR) to more complex, non-linear responses (e.g. SVM), and the bases of the algorithms are fundamentally different: statistical (PLSR), tree-based (Cubist, RF, XGBoost), Gaussian process-based (GPRL and GPRR), and support vector methods. Viscarra Rossel and Behrens (2010) described these algorithms and their implementation in soil spectroscopic modelling.

The average spectra of the replicates from the 56 plot data were used for modelling (Fig. 2). The models were developed using 10-fold cross-validation, and their hyperparameters were optimised by minimising the root mean squared error (RMSE) with either grid search or using the differential evolution optimisation (Price et al., 2006), implemented in the R library DEoptim (Mullen et al., 2011). The optimal number of factors to use in the PLSR was determined using a grid search, implemented in the R library pls (Wehrens and Mevik, 2007). We implemented the SVM using a Gaussian radial basis function in the R library kernlab (Karatzoglou et al., 2004). Its hyperparameter C, which describes the cost or penalty on training accuracy and behaves as a regularisation parameter in the SVM, and γ, which defines the influence of training samples, were optimised using DEoptim. RF was implemented using the R library randomForest (Liaw and Wiener, 2002) and the hyperparameter mtry, which is the number of predictors randomly sampled as candidates at each decision tree split, was optimised with DEoptim. We implemented GPR using the R library kernlab and tested both linear and polynomial kernels. The linear method did not need optimising; however, for the polynomial kernel, we used DEoptim to optimise the degree of the polynomial and scale hyperparameter. The optimisation of hyperparameters in XGBoost, implemented using the R library xgboost, was also performed using DEoptim. The hyperparameters optimised were the number of iterations (nrounds), the learning rate (eta), maximum tree depth (max_depth), the regularisation parameter, which controls overfitting (γ), the number of predictors supplied to each tree (colsample_bytree), the minimum number of instances required in a child node (min_child_weight), and the number of samples (observations) supplied to a tree (subsample). Viscarra Rossel and Webster (2012) described the implementation of Cubist in spectroscopic modelling. Its hyperparameters, the number of committees (c) and neighbours (N), were optimised with DEoptim.

For each soil property, we calculated the mean, maximum, and minimum concordance correlation coefficient (Lin, 1989), ρ_c, of each algorithm and selected only the most accurate and consistent ones to compare the performance of the spectrometers and their combinations. The ρ_c is a unit-invariant coefficient that measures the difference between the measured and estimated values and their deviation from a 45^∘ line of perfect agreement, evaluating both precision and bias. It ranges from −1 to 1, with 1 denoting perfect agreement. The ρ_c was used as the main metric for the assessments because it allows for the comparison across soil properties with different units possible. We used ρ_c<0.65 to denote poor agreement between the measured values of the properties and the estimates, $\mathrm{0.65}\le {\mathit{\rho }}_{\mathrm{c}}<\mathrm{0.8}$ to denote moderate agreement, $\mathrm{0.8}\le {\mathit{\rho }}_{\mathrm{c}}<\mathrm{0.9}$ for substantial agreement, and $\mathrm{0.9}\le {\mathit{\rho }}_{\mathrm{c}}\le \mathrm{1.0}$ for near-perfect agreement. We removed algorithms with an average ρ_c below 0.65 as they performed poorly with most spectrometers and soil properties. Their removal improved the modelling and interpretability of our results and reduced redundancy in the experiments.

2.4.2 Assessment of spectrometers with data from plots

Accuracy

To compare the performance of the different spectrometers, we modelled the 29 soil properties using the five spectrometers and the three spectrometer combinations. We first selected the algorithms yielding the most consistent estimates (see above) for the soil properties and then calculated the average, minimum, and maximum ρ_c across the best algorithms to assess the spectrometers' and the combinations' accuracy for each property (Fig. 2).

To more explicitly quantify the estimation error with the spectrometer or spectrometer combinations that produced the best predictions (i.e. the largest ρ_c), we measured the RMSE, the mean error (ME), and the standard deviation of the error (SDE), which represent the inaccuracy, bias, and imprecision of the estimates, respectively. We note that the inaccuracy (RMSE) embraces both the bias (ME) and the imprecision (SDE) of the analysis so that RMSE² = ME² + SDE². Because RMSE, ME, and SDE are unit-variant, they are not suitable for comparing errors across soil properties with differing units. They were used to quantify the overall inaccuracy (RMSE), the bias (ME), and the imprecision (SDE) of the estimates for each soil property.

Repeatability

We assessed the repeatability of the spectroscopic measurements by calculating the ratio of the difference between the replicates to the mean of the replicates (Fig. 2), defined as

$$\begin{array}{}\text{(1)}& \mathit{\%}\text{Difference}={\displaystyle \frac{|{\mathrm{Rep}}_{\mathrm{a}}-{\mathrm{Rep}}_{\mathrm{b}}|}{\left({\mathrm{Rep}}_{\mathrm{a}}+{\mathrm{Rep}}_{\mathrm{b}}\right)/\mathrm{2}}}\times \mathrm{100},\end{array}$$

where Rep_a and Rep_b represent the two spectroscopic replicate measurements. Repeatability values (% Difference) closer to zero are considered to represent more repeatable spectroscopic measurements.

To assess the impact of a spectrometer's repeatability on the spectroscopic modelling, we also modelled soil properties with each replicate (Rep_a and Rep_b) and calculated the absolute difference in ρ_c, Δρ_c, of the estimates (Fig. 2). We performed the spectroscopic modelling using the best algorithms (see above) and reported the mean, maximum, and minimum Δρ_c for each spectrometer and combination.

2.4.3 Overall assessment of the spectrometers

To assess the overall performance of each spectrometer and combination as a function of their accuracy and repeatability, we derived the index e:

$$\begin{array}{}\text{(2)}& \begin{array}{rl}e=& {\displaystyle \frac{{\sum }_{i=\mathrm{1}}^N\stackrel{\mathrm{‾}}{{\mathit{\rho }}_{\mathrm{c}}^i}}{N}}+{\displaystyle \frac{{\sum }_{i=\mathrm{1}}^N\left(\mathrm{1}-\left({}^{\text{max}}{\mathit{\rho }}_{\mathrm{c}}^i{-}^{\text{min}}{\mathit{\rho }}_{\mathrm{c}}^i\right)\right)}{N}}\\ +& {\displaystyle \frac{{\sum }_{i=\mathrm{1}}^N\left(\mathrm{1}-\stackrel{\mathrm{‾}}{\mathrm{\Delta }{\mathit{\rho }}_{\mathrm{c}}^i}\right)}{N}}\\ +& {\displaystyle \frac{{\sum }_{i=\mathrm{1}}^N\left(\mathrm{1}-\left({}^{\text{max}}\mathrm{\Delta }{\mathit{\rho }}_{\mathrm{c}}^i{-}^{\text{min}}\mathrm{\Delta }{\mathit{\rho }}_{\mathrm{c}}^i\right)\right)}{N}},\end{array}\end{array}$$

where e is the overall performance index, N is the number of soil properties, $\stackrel{\mathrm{‾}}{{\mathit{\rho }}_{\mathrm{c}}^i}$, ${}^{\text{max}}{\mathit{\rho }}_{\mathrm{c}}^i$, and ${}^{\text{min}}{\mathit{\rho }}_{\mathrm{c}}^i$ are the mean, maximum, and minimum ρ_c for the ith property from the best algorithms, and $\stackrel{\mathrm{‾}}{\mathrm{\Delta }{\mathit{\rho }}_{\mathrm{c}}^i}$, ${}^{\text{max}}\mathrm{\Delta }{\mathit{\rho }}_{\mathrm{c}}^i$, and ${}^{\text{min}}\mathrm{\Delta }{\mathit{\rho }}_{\mathrm{c}}^i$ are the mean, maximum, and minimum difference in ρ_c from the modelling with replicate measurements for the ith property with the best algorithms. The first two terms in Eq. (2) assess the overall accuracy and stability of the accuracy when the calibrations are performed using different algorithms. The third and fourth terms assess the effect of the spectrometers' repeatability on the spectroscopic modelling and its stability. All terms range from 0 to 1, and a higher value means better performance.

2.4.4 Assessment on data from subplots

To further verify results and conclusions from the 56 aggregated plot data, we also evaluated the performance of the spectrometers using data from the 280 subplots. When modelling the 280 subplot data (Fig. 2), we used only the algorithm that performed best in modelling the data from the plots. As the subplots originated from within the reference and stockpile plots, we performed the modelling and assessment using 10-fold plot-out cross-validation to prevent the risk of overfitting because soil properties and spectra from a reference or stockpile plot can be correlated.

3 Results

The soil samples varied markedly in their physical, chemical, and biological properties, providing a wide range of values for spectroscopic modelling. The soil samples were highly variable, with wide ranges in their physicochemical and biological properties. They represented soils that were in a good, healthy condition (reference sites) to soils that were degraded (stockpiles). For example, clay content ranged from 4 %–66 %, organic C from 0.19 %–4.3 %, pH 5.12 to 9.1, and microbial activity (CO₂ flux) from 5–140 mg L⁻¹ (Table 3).

Table 3 Summary of the physical (n=5), chemical (n=19), and biological (n=5) properties (mean, SD, minimum, maximum, median, and first and third quartiles) of reference soil and stockpiled topsoil sampled from seven mine sites in Western Australia, as determined by conventional analytical methods.

The reflectance spectra from the different spectrometers and their combinations show features that are characteristic of soil spectra. For example, the feature near 1900 nm (Fig. 3) is due to combination and overtone vibrations of molecular water contained within soil minerals (Viscarra Rossel and Behrens, 2010).

Figure 3 Mean and difference spectra of the spectral replicates of the data from the 56 sampling plots. (a) Average reflectance spectra of the two replicates (Fig. 2). (b) Difference between the two replicates calculated using Eq. (1). (c) Combined average reflectance spectra from (a).

Spectrometer E_350−2500 was more repeatable in the range between 600–2500 nm, with less than 18.7 % difference between the replicates. At wavelengths smaller than 600 nm, however, the difference between the replicates was large (Fig. 3). The repeatability of spectrometers B_1750−2150 (<24.3 %), C_1450−2450 (<20.6 %), and D_1300−2600 (<30.9 %) was similar, with B_1750−2150 and C_1450−2450 being slightly more repeatable than D_1300−2600. In contrast, replicate spectra from A_350−830 were more imprecise in the 350–500 nm region (Fig. 3).

3.1 Assessment of the algorithms on data from plots

PLSR, GPRP, Cubist, and GPRL consistently produced more accurate estimates (mean ρ_c of 0.74, 0.74, 0.71, and 0.69 respectively) of the soil physical, chemical, and biological properties and with less variability (narrower minimum, maximum intervals) between spectrometers than SVM, RF, and XGBoost (mean ρ_c of 0.51, 0.54, and 0.58 respectively) (Fig. 4).

Figure 4 Assessment of the accuracy from different algorithms. The mean ρ_c (points) and range (lines) extend the minimum and maximum values of ρ_c from the five spectrometers and three combinations. The three shaded grey bands indicate the accuracy level, which we classified as poor (ρ_c<0.65), moderate ($\mathrm{0.65}\le {\mathit{\rho }}_{\mathrm{c}}<\mathrm{0.8}$), substantial ($\mathrm{0.8}\le {\mathit{\rho }}_{\mathrm{c}}<\mathrm{0.9}$), and near-perfect ($\mathrm{0.9}\le {\mathit{\rho }}_{\mathrm{c}}\le \mathrm{1.0}$).

Of the soil physical properties, bulk density was most poorly estimated (ρ_c<0.65) (Fig. 4). All of the algorithms performed well for the rest of the physical properties (ρ_c≥0.65). The accuracy of the estimates of the biological properties varied. CO₂ flux with PLSR, Cubist, GPRL, and GPRP produced ρ_c values between 0.65–0.8 (Fig. 4). Estimates of fungal diversity and richness with these algorithms varied, and only PLSR and GPRP produced mean values of ρ_c in the range between 0.65–0.8. All algorithms poorly estimated bacterial richness and diversity (ρ_c<0.65), although the Cubist estimates were markedly better (Fig. 4). The algorithms inaccurately estimated the chemical properties P, Zn, and ammonium nitrogen (ρ_c<0.65). The chemical properties, pH_Ca, pH_W, organic C, K, B, Cu, Fe, exchangeable Ca, Mg, and K, were predicted with substantial accuracy (ρ_c≥0.8) by PLSR, Cubist, GPRP, and GPRL. The variability in the estimates of the exchangeable Ca, Mg, and Al was large compared to other chemical properties, arising from the performance of the different spectrometers (Fig. 4).

3.2 Assessment of spectrometers' accuracy on data from plots

In this section, we assess the different spectrometers and combinations, considering only the best algorithms: PLSR, Cubist, GPRL, and GPRP (see above and Fig. 4). Generally, combining visible and NIR spectrometers produced more accurate estimates of the soil's physical, biological, and chemical properties than only visible or NIR spectrometers (Fig. 5).The visible range spectrometer, A_350−830, accurately estimated soil texture (sand, silt, and clay), but it was the least accurate for estimating chemical properties. The NIR spectrometers estimated most of the soil physical, chemical, and biological properties with moderate or greater accuracy (ρ_c≥0.65). The B_1750−2150 spectrometer, with the narrowest spectral range, produced the least accurate estimates of the soil physical and biological properties (Fig. 5).

Figure 5 Photographs of the miniaturised and portable spectrometers indicating their size (cm). The spectrometer names, spectral ranges, and combinations are colour-coded and correspond with colours of the symbols on the plots. Accuracy of the spectrometers and combinations. The discs show the mean ρ_c, and the range lines are the minimum and maximum values of ρ_c from the best four algorithms (PLSR, Cubist, GPRL, and GPRP). The grey bands indicate the level of accuracy (moderate, substantial, and near-perfect – see caption of Fig. 4).

The A_350−830 spectrometer produced the best estimates of silt content, and its estimates of sand and clay were comparably accurate to those made using instruments that cover the NIR and vis–NIR ranges (Fig. 5). Estimates of sand and silt content using only the NIR range, with the B_1750−2150, C_1450−2450, and D_1300−2600 spectrometers, were the least accurate. Estimates of soil electrical conductivity with the NIR and vis–NIR ranges were similar ($\mathrm{0.8}\le {\mathit{\rho }}_{\mathrm{c}}<\mathrm{0.9}$, Fig. 5) and remarkably better than the visible range. The spectrometers that combine the visible and NIR ranges estimated the soil biological properties better. For instance, the combined $A_{\mathrm{350}-\mathrm{830}}+C_{\mathrm{1450}-\mathrm{2450}}$ and $A_{\mathrm{350}-\mathrm{830}}+D_{\mathrm{1300}-\mathrm{2600}}$ spectrometers produced the most accurate estimate of CO₂; $A_{\mathrm{350}-\mathrm{830}}+B_{\mathrm{1750}-\mathrm{2150}}$ and the portable E_350−2500 spectrometer produced better estimates of fungal richness and diversity, respectively (Fig. 5). Of the soil chemical properties that were estimated with at least moderate accuracy (ρ_c≥0.65), the NIR and vis–NIR range spectrometers, C_1450−2450, D_1300−2600, $A_{\mathrm{350}-\mathrm{830}}+B_{\mathrm{1750}-\mathrm{2150}}$, $A_{\mathrm{350}-\mathrm{830}}+C_{\mathrm{1450}-\mathrm{2450}}$, $A_{\mathrm{350}-\mathrm{830}}+D_{\mathrm{1300}-\mathrm{2600}}$, and E_350−2500, produced estimates with similar accuracy (Fig. 5), whereas the visible range spectrometer, A_350−830, produced inaccurate results.

3.3 Assessment of the spectrometers' repeatability on data from plots

For all of the soil physical, chemical, and biological properties, the E_350−2500 full-range vis–NIR portable spectrometer produced estimates that were generally the most repeatable (lowest mean Δρ_c; Fig. 6). The miniaturised NIR spectrometer (B_1750−2150, C_1450−2450, and D_1300−2600) and combined vis–NIR ($A_{\mathrm{350}-\mathrm{830}}+B_{\mathrm{1750}-\mathrm{2150}}$, $A_{\mathrm{350}-\mathrm{830}}+C_{\mathrm{1450}-\mathrm{2450}}$, and $A_{\mathrm{350}-\mathrm{830}}+D_{\mathrm{1300}-\mathrm{2600}}$) produced similar and repeatable estimates of the soil properties.

Due to the poor repeatability of the A_350−830 measurements (see Fig. 3), the estimates of the soil properties with the spectra from this instrument were the most uncertain, particularly the chemical properties (Fig. 6). For a number of properties that were not precisely estimated with the A_350−830 instrument (e.g. bulk density, S, and Exchangeable Na and Al), combining it with a NIR spectrometer reduced the precision of the estimates from the spectrometer combinations (larger mean Δρ_c). For the soil properties that could not be estimated well with any spectrometer (e.g. bacterial richness and diversity, P, Zn, and ammonium nitrogen with ρ_c<0.65), the precision of the estimates was also poor.

Figure 6 Effect of the repeatability of the spectrometers on the spectroscopic modelling. Δρ_c represents the difference in ρ_c between the algorithms (with PSLR, Cubist, GPRL, and GPRP) of the replicates. The discs show the mean difference, and the lines represent the minimum and maximum of the difference. A small Δρ_c indicates good repeatability.

3.4 Overall assessment of the spectrometers

Spectrometers that cover the visible and NIR ranges were the most accurate and stable as shown by the higher accuracy and accuracy range scores, respectively (Table 4). The $A_{\mathrm{350}-\mathrm{830}}+C_{\mathrm{1750}-\mathrm{2450}}$ and $A_{\mathrm{350}-\mathrm{830}}+D_{\mathrm{1300}-\mathrm{2600}}$ spectrometers were as accurate as the full-range portable E_350−2500 spectrometer. Spectrometers C_1450−2450 and E_350−2500 were the most repeatable, while A_350−830 was the least repeatable (Table 4). The precision of the vis–NIR combinations $A_{\mathrm{350}-\mathrm{830}}+B_{\mathrm{1750}-\mathrm{2150}}$, $A_{\mathrm{350}-\mathrm{830}}+C_{\mathrm{1450}-\mathrm{2450}}$, and $A_{\mathrm{350}-\mathrm{830}}+D_{\mathrm{1300}-\mathrm{2600}}$ was affected by the poor repeatability of A_350−830. Overall, the full-range vis–NIR spectrometer (E_350−2500) had the highest e score (Table 4), followed by NIR spectrometers with wider spectral ranges (C_1450−2450, D_1300−2600) and vis-NIR combinations ($A_{\mathrm{350}-\mathrm{830}}+C_{\mathrm{1450}-\mathrm{2450}}$ and $A_{\mathrm{350}-\mathrm{830}}+D_{\mathrm{1300}-\mathrm{2600}}$).

Table 4 Overall assessment of the spectrometers and combinations. Columns show the accuracy and the stability (represented by the range of the accuracy) of the spectrometers when modelling with the different algorithms (PSLR, Cubist, GPRL, and GPRP) as well as the effect of the spectrometers' repeatability on the modelling and its stability. They correspond to the first, second, third, and fourth term in Eq. (2), respectively.

For each soil property, we derived the performance score, e, for all the spectrometers and combinations. The evaluation statistics of the spectrometer or combination with the largest e are shown in Table 5. The miniaturised spectrometers and combinations estimated 24 of the 29 soil properties with moderate or better accuracy (ρ_c≥0.65), except for P, Zn, ammonium nitrogen, and bacterial properties. The inaccuracy of the soil property estimates, quantified with the RMSE, was largely due to imprecision (SDE) and not bias (ME).

The visible range spectrometer A_350−830 had the largest e for sand and silt, with ρ_c≥0.90 and $\mathrm{0.80}\le {\mathit{\rho }}_{\mathrm{c}}\le \mathrm{0.90}$, respectively (Table 5). The individual NIR spectrometers performed well on many chemical and some of the biological properties. The C_1450−2450 spectrometer alone predicted one soil physical property, six chemical properties, and one biological property with moderate or better accuracy (ρ_c≥0.65). Some soil properties (e.g. clay, electrical conductivity, pH_Ca, K, Mn, exchangeable Na, K, and Al, and fungal diversity) were accurately estimated when the visible spectrometer was combined with a NIR spectrometer, for instance, the combined $A_{\mathrm{350}-\mathrm{830}}+D_{\mathrm{1300}-\mathrm{2600}}$ spectrometers.

Table 5 Evaluation statistics for the miniaturised spectrometers and combinations showing the highest e score for individual soil properties.

3.5 Spectroscopic modelling with PLSR and 10-fold plot-out cross-validation on data from subplots

Compared to the other algorithms tested, PLSR generally produced more accurate estimates of the soil properties tested (see Fig. 4), so we used it for modelling the 280 data from the subplots (see Methods section, “Validation on data from subplots”). Since the A_350−830 and B_1750−2150 spectrometers generally produced the least accurate estimates (Fig. 5) and had the smallest e score (Table 4), we did not use them in this modelling. The 10-fold plot-out cross-validations of the soil properties from subplots (Fig. 7) were similar or slightly more conservative compared to those of the 10-fold cross-validation of the data from plots (Fig. 5), suggesting that our evaluation of the spectrometers is reliable.

Figure 7 The 10-fold plot-out cross-validation using PLSR with data from subplots.

The accuracy of the estimates with the combined $A_{\mathrm{350}-\mathrm{830}}+C_{\mathrm{1450}-\mathrm{2450}}$ and $A_{\mathrm{350}-\mathrm{830}}+D_{\mathrm{1300}-\mathrm{2600}}$ spectrometers was similar to the full-range portable E_350−2500 spectrometer and better than the combined $A_{\mathrm{350}-\mathrm{830}}+B_{\mathrm{1750}-\mathrm{2150}}$ spectrometer (Fig. 7). The C_1450−2450 and D_1300−2600 spectrometers produced estimates with similar accuracy, but their estimates were less accurate than those from the $A_{\mathrm{350}-\mathrm{830}}+C_{\mathrm{1450}-\mathrm{2450}}$ and $A_{\mathrm{350}-\mathrm{830}}+D_{\mathrm{1300}-\mathrm{2600}}$ spectrometers, respectively.

4 Discussion

4.1 Mine site soil rehabilitation requires cost-effective soil property measurements

Mining is vital to economic development in many countries. In Australia, for example, mining and energy exports are forecast to be worth around AU 288 billion in 2020/21, and over 184 billion of this from WA alone (Department of Industry, Science, Energy and Resources, 2021). However, the economic benefits of mining come at an environmental cost, and the collective footprint of mining in Australia is expected to exceed 4000 km² by 2050 (EPA, 2014), and globally, it is currently 57 000 km² and has been increasing at an unparalleled rate in the last decade. There is little historical evidence of capacity to effectively restore land at this scale (EPA, 2013), and regulatory bodies have urged the mining industry to engage in restoration science (EPA, 2014).

To rehabilitate and restore biodiverse, resilient ecosystems post-mining, we must first measure and diagnose, then reinstate and monitor the health of the soils. However, a lack of rapid, quantitative methods for assessing and monitoring soil properties that are indicators of soil health may at least partially underpin the continuing failure to deliver effective and cost-efficient restoration outcomes following mining. In this context, our results are encouraging because they present an opportunity for establishing a science-based diagnostic capacity to rapidly and cost-effectively estimate soil properties that are key for diagnosing soil health.

4.2 Miniaturised visible and NIR spectrometers can estimate key soil health indicators

We used miniaturised spectrometers to estimate 29 soil physical, chemical, and biological soil properties. Amongst these, soil organic carbon, K, P, pH, EC, B, Cu, Zn, Mn, Fe, Ca, Mg, Na, Al, sand, silt, clay, soil respiration (CO₂ ), available N (ammonium and nitrate), and BD are tier 1 soil health indicators in routine soil testing programs (Karlen et al., 2019; Bünemann et al., 2018). Although less commonly included in soil health assessment frameworks, microbial community composition is an under-utilised yet valuable metric for soil health assessment (Fierer et al., 2021). Changes in soil microbial communities are associated with processes that are important for soil health, such as changes in P availability (Hermans et al., 2017), soil pH (Delgado-Baquerizo et al., 2018), labile organic carbon pools (Ramírez et al., 2020), and soil moisture levels (Isobe et al., 2020). Our aim was not to derive detailed interpretations of the soil health indicators or an overall soil health index or score. Instead, we tested 29 soil properties commonly used as key indicators of soil health and used the spectra from a range of miniaturised and portable spectrometers to model them. The spectroscopic models accurately estimated 24 out of 29 soil properties (Table 5). Therefore, the spectra from the miniaturised spectrometers can accurately estimate soil properties that are key indicators of soil health. These estimates can could then inform the assessment and diagnosis of soil health. These findings suggest using miniaturised spectrometers to develop a cost-effective soil health diagnostic capacity for mine site soil rehabilitation. In a future study, we might investigate the derivation of a soil health index based on the spectra or the estimated soil properties.

Some soil properties (bacterial richness and diversity, P, Zn, and ammonium nitrogen) could not be estimated well with any of the spectrometers or combinations (Fig. 5). Therefore, it might be necessary to investigate other methods to measure them. Although we could estimate 24 soil properties with moderate or greater accuracy (ρ_c≥0.65), the estimates are outcomes from empirical models that possess error (Table 5). However, the advantage of the spectroscopic method, compared to the more accurate conventional analytical measurements, is that spectroscopy is rapid and cost-effective (Li et al., 2022), allowing for many more (spatial and temporal) measurements, which on the whole might serve to better assess and monitor soil health. Another advantage of spectroscopy is that a single spectrum can be used to estimate many soil properties. These make soil spectroscopy well suited for large-scale soil rehabilitation where many data are needed but are too expensive and time-consuming to obtain using conventional soil analysis.

The accuracy of the estimates of electrical conductivity and most of the soil chemical properties from the A_350−830 spectrometer was markedly poorer than the NIR spectrometers (Fig. 5), indicating that the 350–830 nm range does not hold sufficient chemical information to produce suitable models for estimating those soil properties. The poor repeatability of the A_350−830 spectrometer’s measurements in the 350–500 nm range (Fig. 3) also affected the precision of the spectrometer combinations (reduced repeatability of the combinations in Table 4). With a more repeatable visible spectrometer, the spectrometer performance of the combined spectrometers would improve.

4.3 Soil assessment with the miniaturised spectrometers

Some studies have evaluated miniaturised NIR spectrometers for characterising soil properties (Tang et al., 2020; Ng et al., 2020; Sharififar et al., 2019) in agricultural and natural environments. However, our study presents a comprehensive assessment. We tested a diverse range of soil properties considered key indicators of soil health using many spectrometers (one visible, three NIR, three combinations, and a portable spectrometer) and seven statistical and machine learning algorithms; we evaluated the repeatability of the measurements and their effect on the models' estimates and the accuracy of the models. Our study also extends the application of miniaturise spectrometers in mine site soil rehabilitation.

How can the miniaturised spectrometers with coarser resolutions and narrower spectral ranges produce nearly as good or better results as the full-range, higher-resolution sensor? For different reasons. First, soil vis–NIR spectra are non-specific and highly collinear (Stenberg et al., 2010). Although the small spectrometers have restricted wavelength ranges, a well-selected spectral range can hold information on overtones and combination bands of important organic and mineral constituents that enable the development of accurate models. For example, spectrometer B_1750−2150, with a very narrow range, could estimate many soil properties with moderate or greater accuracy (ρ_c≥0.65, Fig. 5). Table 5 shows the spectrometers with varying spectral ranges that could adequately estimate each soil property. Second, absorptions due to soil constituents in the vis–NIR range are broad (Viscarra Rossel and Behrens, 2010), and coarse spectral resolutions, like those of the miniature spectrometers (Table 2), are unlikely to affect the modelling, compared to the portable vis–NIR spectrometer with a finer spectral resolution.

4.4 Miniaturised soil spectroscopy can support soil rehabilitation

The portability and affordability (see Table 2) of the miniaturised spectrometers enable the cost-effective acquisition of soil information at greater temporal and spatial resolutions than conventional laboratory-based methods. Although spectroscopy produces less precise measures of soil properties than conventional laboratory analyses, it enables many more rapid and cost-effective measurements at the appropriate spatial and temporal resolution for rehabilitation and ecological restoration. Practitioners can then effectively identify the need for early interventions to establish positive soil health trajectories. Spectroscopy could also facilitate the evaluation of soil degradation, more timely identification and remediation of ecologically hostile conditions, and more effective monitoring of the change in soil properties in response to restoration activities. An additional significance of the miniaturised visible and NIR spectrometers is that they are, even in combination, much cheaper than the full-range portable instrument (Table 2). Because they are cost-effective, compact, and lightweight, they are being used in many diverse applications (Giussani et al., 2022; Pu et al., 2021).

Together with other sensors and environmental data, development and further testing of the miniaturised visible and NIR spectrometers could provide the mining industry and restoration practitioners with a rapid and cost-efficient methodology for diagnosing, assessing, and monitoring soil health. The information gained would ensure that soil management, whether in rehabilitation and restoration, is underpinned by quantitative information for evidence-based decision-making. In addition, the ability to reliably assess a wide range of soil properties that are key indicators of soil health enables rapid identification of when intervention is required, which could help deliver significant economic and environmental outcomes.

5 Conclusions

Achieving desired outcomes from rehabilitation and ecological restoration relies on characterising soil properties that affect soil health. Therefore, developing efficient, reliable, and cost-effective methods for measuring and monitoring soil properties that can indicate soil health is essential. We compared four miniaturised spectrometers (one visible and three NIR) and three vis + NIR combinations to a research-grade (benchmark), full-range vis–NIR instrument, which is an order of magnitude more expensive. Seven statistical and machine learning algorithms were used for modelling 29 soil physical, chemical, and biological properties, key indicators of soil health, to assess the spectrometers' predictive performance as a function of accuracy and repeatability. Four algorithms, PLSR, Cubist, GPRL, and GPRP, performed consistently well with the different spectrometers. The visible spectrometer was remarkably accurate for estimating sand, silt, and clay contents, and the NIR spectrometers could estimate most chemical, physical, and biological properties with moderate or greater accuracy (ρ_c≥0.65). However, none of the spectrometers could estimate bacterial richness or diversity. The visible sensor was the least repeatable, which affected its estimates and those with the combined spectrometers. The combined small vis + NIR spectrometers performed similarly to the benchmark spectrometer, and the miniaturised spectrometers and combinations could estimate 24 of the 29 soil properties with moderate or greater accuracy. Soil spectroscopy with these tiny sensors is cost-effective. It enables the collection of many more measurements at the required spatial and temporal resolutions, improving soil health diagnosis and ecologic rehabilitation and restoration. Thus, the information gained could inform decision-making to deliver positive environmental and economic outcomes.