: This paper identifies and evaluates methods for measuring the benefit or utility of large-scale training simulations (LSTSs) The intended audience is those people who are directly responsible for assessing the training effectiveness of LSTSs We examine literature related to the measurement of LSTS training effectiveness INCLUDING GUIDANCE PROVIDED BY THE INDIVIDUAL MILITARY Services and the Office of the Secretary of Defense (OSD), the body of research on performance measurement, and research practices related to experimental design and analysis We then synthesize the findings from these literature sources to provide seven recommendations for performing training effectiveness analyses of LSTSs: (1) identify specific measurement issues, (2) devise a measurement plan that specifies performance measurement and research design, (3) use valid and reliable performance measures, (4) obtain the most valid measure of proficiency possible within the constraints of the evaluation, (5) to the extent possible and reasonable, impose experimental control on the research situation, (6) use analytic models in early stages and continue using these models throughout system development, (7) and include user reactions as an adjunct to performance data and analysis

On Measuring the Effectiveness of Large-Scale Training Simulations

The above real-time training (ARTT) concept is an approach to teaching high-performance skills. ARTT refers to a training paradigm that places the operator in a simulated environment that functions at faster than normal time. It represents a departure from the intuitive, but not often supported, feeling that the best practice is determined by the training environment with the highest fidelity. This approach is hypothesized to provide greater 'transfer value' per simulation trial, by incorporating training techniques and instructional features into the simulator. Two related experiments are discussed. In the first, 25 naive male subjects performed three tank gunnery tasks on a simulator under varying levels of time acceleration (i.e., 1.0x, 1.6x, 2.0x, sequential, and mixed). They were then transferred to a standard (1.0x) condition for testing. Every accelerated condition or combination of conditions produced better training and transfer than the standard condition. Most effective was the presentation of trials at 1.0x, 1.6x, and 2.0x in a random order during training. Overall, the best ARTT group scored about 50 percent higher and trained in 25 percent less time compared to the real-time control group. In the second experiment, 24 mission-capable F-16 pilots performed three tasks on a part-task F-16A flight simulator under varying levels of time compression (i.e., 1.0x, 1.5x, 2.0x, and random). All subjects were then tested in a real-time environment. The emergency procedure (EP) task results showed increased accuracy for the ARTT groups. In testing (transfer), the ARTT groups not only performed the EP more accurately, but dealt with a simultaneous enemy significantly better than a real-time control group. Although the findings on an air combat maneuvering task and stern conversion task were mixed, most measures indicated that the ARTT groups performed better and faster than a real-time control group. Other implications for ARTT are discussed along with future research directions.

Teaching high-performance skills using above-real-time training

: This study was performed to determine (1) whether performance improves during part-task tank gunnery training on TOPGUN and VIGS devices; (2) whether TOPGUN performance transfers to VIGS performance, and vice versa; (3) whether training performance on VIGS and TOPGUN transfers to gunnery performance an a high-fidelity trainer--the Institutional Conduct-of-Fire Trainer (ICOFT); (4) which sequence of training (TOPGUN-VIGS or VIGS-TOPGUN) shows better transfer to ICOFT; and (5) whether selected aptitude, ability, motivational, and demographic measures predict TOPGUN, VIGS, and ICOFT performance.

An Analysis of Skill Transfer for Tank Gunnery Performance Using TOPGUN, VIGS, and ICOFT Trainers

: This literature review was conducted to help establish a ground for future research and development to improve Reserve Component (RC) training. Using computer searches and bibliographies obtained from various information services, documents published over the last 10 years were selected. The literature shows that the RC faces training challenges quite distinct from those found in the Active Component (AC). The most prominent challenges are severely limited time for training, the geographical dispersion of units, and the reassignment inflexibility of parttime soldiers. The literature revealed deficiencies in five major areas: soldier availability, prerequisite aptitudes and skills, learning motivation, time to conduct training, and training resources at the local unit level. Numerous suggestions and solutions to improve RC training are proposed in the literature and summarized in this report. Keywords: Inactive duty training, Army National Guard, U.S. Army Reserve, Training technology.

/pdf/review-of-reserve-component-training-problems-and-solutions-5dkg4izzcn.pdf

Review of Reserve Component Training: Problems and Solutions

: Research was conducted to determine if performance on the first run of Tank Table VIII can be predicted from amount and level of training on the Unit Conduct-of-Fire Training (U-COFT) and from crew turbulence data During the first phase of the research, intercorrelations were computed between predictor variables and Tables VIII performance measures obtained from 77 tank crews Two predictor variables from U-COFT training correlated significantly with Table VIII performance--crew reticle air level and TC reticle aim level Neither variable was able to predict which crews would qualify on Table VIII, but the composite of crew reticle air and time in crew was able to predict which crews would qualify This finding was not supported by the results based on 136 crews examined during Phase II Since gunnery training was curtailed during Phase I but not during Phase II, the Phase I results may be applicable to future training situations in which resource constraints cause to be curtailed (sdw)

/pdf/predicting-first-run-gunnery-performance-on-tank-table-viii-18r25xunwp.pdf

Predicting First-Run Gunnery Performance on Tank Table VIII

: Tank Gunnery training devices are designed to decrease costs and other resources required for training. To realize resource savings and, at the same time, maintain desired proficiency, cost-effective tradeoffs must be made between device training and on-tank training. To determine tradeoff specifications, experimental and nonexperimental methods were reviewed, critiqued, and synthesized into a set of recommendations. A surrogate method, simulated transfer, which uses judgments from subject matter experts, was modified for gunnery training problems. Also delineated were nonlinear models needed to guide analysis of learning and tradeoff data. Researchers identified an Army National Guard setting for testing the nonexperimental and simulated transfer research methods. Research focused on the Guard Unit Army Device Full- Crew Interactive Simulation Trainer, Armor (GUARD FIST I). Proficiency ratings, collected on all gunnery training events, showed some ability to predict relative performance from one training event to another, but mean ratings varied dramatically between events, suggesting that the interpretation of the rating scale shifted between raters and events. Amount of practice was negatively correlated with Table VIII, apparently because of a training stategy that gave priority in training time to crews having lower proficiency.

/pdf/methods-for-determining-resource-and-proficiency-tradeoffs-4241c6lbc4.pdf

Methods for Determining Resource and Proficiency Tradeoffs among Alternative Tank Gunnery Training Methods

: The first volume of this report (ARI Technical Report 783) presented the analytic rationale for determining the requirements of a device-based program for training and testing armor skills. The resulting concept indicated that it was feasible to use existing devices to train and test much of the domain of tank gunnery in an integrated fashion. However, there were gaps in the program where devices did not support substantial portions of the domain. Also, there were cases where training on basic skills and knowledges was not integrated into training on higher level skills. It was concluded that on-tank experience at both the beginning and advanced stages of training is necessary to train and test the entire domain of gunnery. This second volume of the report details the results that support the research activities described above. These detailed results are presented as a series of appendixes that support the following research activities: (a) definition of the domain of gunnery in terms of individual conditions and actions; (b) identification of gunnery performance deficiencies through an analysis of Directorate of Evaluation and Standardization (DOES) performance data, a review of the research literature, and an analysis of Table VIII performance at Grafenwohr; (c) identification of training and testing objectives through a hierarchical analysis of the gunnery domain; (d) evaluation of gunnery devices with respect to their fidelity features, instructional features, and testing features; and (e) specification of the training and testing program by a detailed listing of objectives within instructional units. Keywords: Gunnery trainers; Tank crews; Training devices; Army training.

Requirements for a Device-Based Training and Testing Program for M1 Gunnery. Volume 2. Detailed Analyses and Results

: This report is concerned with several interrelated questions that have arisen because performance requirements and training device capabilities are both expanding. The project began with an analysis of the domain of gunnery stimulus conditions and behaviors. This analysis pulled together information from a variety of sources and organized it into a more comprehensive description of M1 gunnery procedural elements than available from any one existing source. The tactical gunnery domain is described by two lists: (a) a list of tank gunnery conditions and (b) a list of tank gunnery behaviors. These two lists are not independent, but contain intentional redundancies. That is, where conditions create qualitatively different gunnery behaviors (e.g., single target versus multiple targets), the behaviors are segregated as separate activities. The gunnery domain description was used to define training objectives and to guide evaluations of the four gunnery devices for both training and testing. Devices included the Videodisc Interactive Gunnery Simulator (VIGS), the arcade-type TopGun device, the Unit Conduct-of-Fire Training (U-COFT), and the Simulated Networking (SIMNET) battle simulation system.

Requirements for a Device-Based Training and Testing Program for M1 Gunnery. Volume 1. Rationale and Summary of Results

: This report describes the development of a method for selecting combat tasks for training based on their criticality in determining the successful accomplishment of the unit mission in which they are performed. Comparisons are made between this method for assessing task criticality and the approach recommended in the Instructional Systems Development (ISD) model. One of the major problems experienced by training developers is the lack of sufficient time and resources to train all of the tasks that are performed by soldiers in a particular duty position. Currently the Army trains only those tasks that can be handled within the available time and resource constraints, but to include those tasks that are most critical to mission success. Although this approach would prevent soldiers from learning all of the tasks performed in an MOS, careful selection of tasks could keep the impact on unit performance to a minimum. Ideally, tasks that are not learned during institutional training could be learned later during unit training or learned informally while on the job.

http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA160544&Location=U2&doc=GetTRDoc.pdf

Measurement of the Criticality of Combat Tasks

: Tank gunnery training devices are designed to decrease the costs and other resources required for training. To realize the potential resource savings of devices and, at the same time, maintain desired proficiency, cost-effective tradeoffs must be specified for device training and on-tank training. This report presents methods for deriving and using these specifications. In addition, the report summarizes the support requirements for obtaining the data necessary for using the methods specified. The recommended approach is to obtain the data from actual performance-based gunnery data. At the same time, it was recognized that this may not be feasible. Because of this difficulty, an alternative judgment-based method, called simulated transfer, is offered to generate surrogate performance data.

R G Hoffman

Papers

Methods for Determining Resource and Proficiency Tradeoffs among Alternative Tank Gunnery Training Methods

Requirements for a Device-Based Training and Testing Program for M1 Gunnery. Volume 2. Detailed Analyses and Results

Requirements for a Device-Based Training and Testing Program for M1 Gunnery. Volume 1. Rationale and Summary of Results

Measurement of the Criticality of Combat Tasks

A User's Introduction to Determining Cost-Effective Tradeoffs among Tank Gunnery Training Methods