This article provides an integrative review of the literature on judgment-based predictions of performance time, often described as task duration predictions in psychology and as expert-based effort estimation in engineering and management science. We summarize results on the characteristics of performance time predictions, processes and strategies, the influence of task characteristics and contextual factors, and the relations between estimates and characteristics of the estimator. Although dependent on the type of study and the level of analysis, underestimation was more frequently reported than overestimation in studies from the engineering and management literature. However, this was not the case in studies from the psychology literature. Our summaries challenge earlier results regarding the effects of factors such as complexity/difficulty and experience. We also question the recurrent finding that small tasks are overestimated and large tasks are underestimated, as this to some extent can be a statistical artifact caused by random error. Several other influences on predictions are identified and discussed. These include various types of anchoring effects, performance and accuracy incentives, task decomposition, request formats, group estimation, revisions of initial ideal or incomplete estimates, level of abstraction, and superficial cues. We summarize similarities and differences between performance time predictions (e.g., number of work hours) and completion time predictions (e.g., delivery dates) because many studies fail to distinguish between these 2 types of predictions. Finally, we discuss methodological issues in time prediction research and implications for research and application.

From origami to software development: a review of studies on judgment-based predictions of performance time.

ion level. It is especially useful in the case of expert-based estimation, where it is easier for experts to embrace and estimate smaller pieces of project work. Moreover, the increased level of detail during estimation—for instance, by breaking down software products and processes—implies higher transparency of estimates. In practice, there is a good chance that the bottom estimates would be mixed below and above the actual effort. As a consequence, estimation errors at the bottom level will cancel each other out, resulting in smaller estimation error than if a top-down approach were used. This phenomenon is related to the mathematical law of large numbers. However, the more granular the individual estimates, the more time-consuming the overall estimation process becomes. In industrial practice, a top-down strategy usually provides reasonably accurate estimates at relatively low overhead and without too much technical expertise. Although bottom-up estimation usually provides more accurate estimates, it requires the estimators involved to have expertise regarding the bottom activities and related product components that they estimate directly. In principle, applying bottom-up estimation pays off when the decomposed tasks can be estimated more accurately than the whole task. For instance, a bottom-up strategy proved to provide better results when applied to high-uncertainty or complex estimation tasks, which are usually underestimated when considered as a whole. Furthermore, it is often easy to forget activities and/or underestimate the degree of unexpected events, which leads to underestimation of total effort. However, from the mathematical point of view (law of large numbers mentioned), dividing the project into smaller work packages provides better data for estimation and reduces overall estimation error. Experiences presented by Jørgensen (2004b) suggest that in the context of expert-based estimation, software companies should apply a bottom-up strategy unless the estimators have experience from, or access to, very similar projects. In the context of estimation based on human judgment, typical threats of individual and group estimation should be considered. Refer to Sect. 6.4 for an overview of the strengths and weaknesses of estimation based on human judgment.

Software Project Effort Estimation

Studies in laboratory settings report that software development effort estimates can be strongly affected by effort-irrelevant and misleading information. To increase our knowledge about the importance of these effects in field settings, we paid 46 outsourcing companies from various countries to estimate the required effort of the same five software development projects. The companies were allocated randomly to either the original requirement specification or a manipulated version of the original requirement specification. The manipulations were as follows: 1) reduced length of requirement specification with no change of content, 2) information about the low effort spent on the development of the old system to be replaced, 3) information about the client's unrealistic expectations about low cost, and 4) a restriction of a short development period with start up a few months ahead. We found that the effect sizes in the field settings were much smaller than those found for similar manipulations in laboratory settings. Our findings suggest that we should be careful about generalizing to field settings the effect sizes found in laboratory settings. While laboratory settings can be useful to demonstrate the existence of an effect and better understand it, field studies may be needed to study the size and importance of these effects.

The Impact of Irrelevant and Misleading Information on Software Development Effort Estimates: A Randomized Controlled Field Experiment

The effects of request formats on judgment-based effort estimation

An important task for managers in information technology (IT) service settings is the judgment of service performance. The complex and intangible nature of IT services, however, renders this task especially difficult. We use a sample of 85 outsourced software development projects to test for the presence of the "input bias," which is defined as the systematic misuse of nondiagnostic input information in forming managerial judgments of outcomes. The service outcome we examine is process performance. The diagnostic inputs are given by objective performance metrics based on the final cost and duration of completed projects, whereas the nondiagnostic inputs are risk anticipations formed by managers prior to the start of the project. We find strong evidence of the input bias, which leads managers' subjective assessments to diverge considerably from objective outcomes, and that it is moderated by contract type. Our study contributes to better service management by improving our understanding of managers' judgments of service performance and how these judgments are formed.

Managers' Judgments of Performance in IT Services Outsourcing

Software projects average about 30 percent accuracy in effort estimation.1 Expecting highly accurate effort estimates might be unrealistic because software development projects are inherently uncertain. Nevertheless, software professionals' tendency toward overly optimistic estimates and their high level of estimation inconsistency suggest potential for improving effort estimation processes.

S. Grimstad

Papers

Avoiding Irrelevant and Misleading Information When Estimating Development Effort