Although the scientific peer review process is crucial to distributing research investments, little has been reported about the decision-making processes used by reviewers. One key attribute likely to be important for decision-making is reviewer expertise. Recent data from an experimental blinded review utilizing a direct measure of expertise has found that closer intellectual distances between applicant and reviewer lead to harsher evaluations, possibly suggesting that information is differentially sampled across subject-matter expertise levels and across information type (e.g. strengths or weaknesses). However, social and professional networks have been suggested to play a role in reviewer scoring. In an effort to test whether this result can be replicated in a real-world unblinded study utilizing self-assessed reviewer expertise, we conducted a retrospective multi-level regression analysis of 1,450 individual unblinded evaluations of 725 biomedical research funding applications by 1,044 reviewers. Despite the large variability in the scoring data, the results are largely confirmatory of work from blinded reviews, by which a linear relationship between reviewer expertise and their evaluations was observed-reviewers with higher levels of self-assessed expertise tended to be harsher in their evaluations. However, we also found that reviewer and applicant seniority could influence this relationship, suggesting social networks could have subtle influences on reviewer scoring. Overall, these results highlight the need to explore how reviewers utilize their expertise to gather and weight information from the application in making their evaluations.

background

/pdf/the-influence-of-peer-reviewer-expertise-on-the-evaluation-2czp9tfwoe.pdf

4 / Pragmatic Fairness: Customary Rules of Deliberation

Citations

The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications

References

The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications

Related Papers (5)

Dynamics of Reasonableness and Fairness in a Pluralistic Legal System: Perspectives from Adat, Islamic and Civil Inheritance Law

Varieties of Fairness and Procedural Cultures: A Comparative Discussion

Fairness and Its Metric in International Criminal Procedure

Demystifying Legal Reasoning: Ordinary Reason Applied to Law: Natural Reasoning and Deduction from Rules

The Rule 11 Studies and Civil Rights Cases: An Inquiry into the Neutrality of Procedural Rules