Preprocessing methods for near-infrared spectrum calibration

TLDR: Using the evaluation framework, appropriate schemes can be found for several datasets, reducing the root mean square error of prediction (RMSEP) by 50%–60% compared with using the raw spectrum.

Abstract: Spectrum preprocessing is an essential component in the near‐infrared (NIR) calibration. However, it has mostly been configured arbitrarily in the literature and calibration applications. In this paper, a systematic evaluation framework was proposed to quantify the effect of preprocessing, where repeated cross‐validation and evaluation are involved. As many as 108 preprocessing schemes were gathered from the literature and were tested on 26 different NIR calibration problems. Using the evaluation framework, appropriate schemes can be found for several datasets, reducing the root mean square error of prediction (RMSEP) by 50%–60% compared with using the raw spectrum. However, the influence of preprocessing is highly data‐dependent, and no universal solution could be found. Taking the effectiveness and correlation into consideration, Savitzky‐Golay (SG), SG1D, and SG1D + vector normalization (VN)(/standard normal variate [SNV]) are worth testing first. Nevertheless, the heterogeneity at both the dataset level and sample level demonstrated the necessity of a complete evaluation. Our scripts are available at https://github.com/jiaoyiping630/spectrum-preprocessing.