Improving manuscript evaluation procedures

TLDR: Convention program committees face a similar problem in selecting papers for presentation, although the published complaints usually focus exclusively on frustrations encountered in dealings with the journals.

Abstract: THE chronic discontent of both journal editors and contributors of articles with existing procedures for evaluating scientific papers is well documented. Convention program committees face a similar problem in selecting papers for presentation (McReynolds, 1971), although the published complaints usually focus exclusively on frustrations encountered in dealings with the journals. Rodman (1970), writing from the perspective of the journal editor, described the heavy responsibility of the "gatekeeper" role. In his term as editor of Social Problems, Rodman rejected about onefourth of the manuscripts received without passing them on to referees. Among those passed on to reviewers, if two referees read the same paper, they frequently disagreed on its merit. Moreover, when manuscripts were rejected, considerable time and energy were expended in "cooling out" the irate authors who questioned editorial judgment. The pressures generated for the editor are evident by Rodman's view of the job.