How reliable are self-report measures of mileage, violations and crashes?

TLDR: In this article, the reliability of self-reported driver mileage, violations, and crashes was evaluated using test-retest reliability, and the correlation between self reports of crashes in different time periods was found to be much larger than expected in one case, indicating a report bias.

About: This article is published in Safety Science.The article was published on 2015-07-01 and is currently open access. It has received 39 citations till now.

Figures

Table 4: Calculations of the differences in numbers of crashes reported in two waves of questionnaires, when the time periods for reporting were overlapping. Shown are, first, the number of subjects, then the actual numbers of crashes reported. Thereafter a calculation of what the expected, highest possible, change in reporting would be, based upon the numbers in each wave and the difference in time period for reporting. Finally, the actual differences found.

Table 3: Results for the effect of exposure on negative driving events. Shown are the number for respondents, and the correlations between mileage and the dependent variables number of crashes and penalty points, in each sample and wave.

Table 2: Descriptive data for the samples. Shown are the number of respondents, the time difference between self-reports (time 1 and time 2), mileage per month (time 1 and time 2), and the mean and standard deviation of number of crashes and penalty points in the reporting periods (time 1 and time 2).

Table 1: Descriptive data for the samples. Shown are the number of respondents, means and standard deviations for age (in years) and experience (years since licensing), and the percent of men in the sample.

Table 5: The correlations between self-reports of the same variables in different waves. Shown (in rows) are the number of respondents, the overlap between report periods, the length of the report periods in years, the time elapsed between questionnaire waves, and the associations between the reports for mileage, points and crashes, and variants of these per mile. Also, the expected correlation between reports of crashes is shown. For the YDS and YDS Control, this expected correlation was based upon the means of the accident variables, while for the SS and RLS it was the degree of overlap between reporting periods which yielded these values.

Frequently asked questions

Q1. What are the contributions in "How reliable are self-report measures of mileage, violations and crashes?" ?

The use of self-reported driver mileage, violations and crashes is very popular in traffic safety research, but their validity has been questioned. Three mechanisms might influence reliability in self report ; actual changes in the variable over time, stable systematic reporting bias, and random error. The results for crashes were compared to expected correlations, calculated from the error introduced by the non-overlapping periods and the variable means. The correlation between self reports of crashes in different time periods was found to be much larger than expected in one case, indicating a report bias, while the other correlation agreed with the predicted value. 

Q2. What is the standard method for reliability measurements?

The standard method for reliability measurements is to calculate the correlation between the results from different data collection occasions (Anastasi, 1988). 

Q3. What is the effect of the memory effect on the reliability of self-reported crashes?

If the memory effect is unsystematic i.e. not due to poor recall for some drivers, but the result of environmental circumstances, this should have a detrimental effect on the between-measurements reliability of self-reported crashes. 

Q4. How was the correlation calculated for the overlapping time periods?

for the overlapping time periods samples (SS and RLS) it was possible to calculate an expected lowest correlation which took the natural change in the number of crashes over time into account, under the assumption that crashes were evenly distributed over time. 

Q5. What is the effect of the e-learning based driver education on traffic offenders?

Drivers committing relevant violations in the areas implementing these e-learning based driver offender schemes could choose to pay a fine and receive penalty points on their driving license (except the seatbelt offenders), or pay for and complete the e-learning course on traffic safety related to their offence. 

Q6. What is the reason why the participants were more vulnerable to providing unreliable self reports?

as some of the participants were rather young, it is possible that they were more vulnerable to providing unreliable self reports due to specific cognitive biases about their driving skills and crash risk (McKenna, Stanier and Lewis, 1991) or memory distortion due to the time period between the traffic event and subsequent recall. 

Q7. What is the relevant study for mileage?

For mileage, the most relevant study was authored by Alonso et al. (2006), who found that self-reports correlated .64 over a period of ten months (other tests indirectly indicate low reliability due to computational difficulties, where people give different estimates due to the time period used, see the review in af Wåhlberg, 2009). 

Q8. How can an expected correlation be calculated?

anexpected correlation can be calculated using the regression equation for the correlation between time period (squared) correlation and the mean of crashes reported. 

Q9. What are the two variables that are commonly used as outcome parameters?

Turning back to traffic safety research, there are two variables that are commonly used as outcome parameters; crashes and violations (also called offences, citations, endorsements, penalties etc.). 

Q10. What effects did the previous studies have on the reliability of self reported crashes?

In previous tests of the reliability of self reported crashes, violations and mileage over time, no effort seems to have been made to disentangle the various possible effects involved; reporting bias, random error (mostly memory problems) and natural changes in the values reported (e.g. Arthur, 1991). 

Q11. How does the report bias affect the reliability of a crash?

report bias has been identified by the use of a lie scale and comparisons of effects versus self-reported and recorded crash data (af Wåhlberg et al., 2010; af Wåhlberg, 2011a).