The power of feedback.

Pedagosko i psiholosko testiranje i procjenjivanje spadaju među najvažnije doprinose znanosti o ponasanju nasem drustvu i pružaju temeljna i znacajna poboljsanja u odnosu na ranije postupke. Iako se ne može ustvrditi da su svi testovi dovoljno usavrseni niti da su sva testiranja razborita i korisna, postoji velika kolicina informacija koje ukazuju na ucinkovitost kvalitetnih instrumenata u onim situacijama u kojima je njihovo koristenje potkrijepljeno validacijskim podacima. Pravilna upotreba testova može dovesti do boljih odluka o pojedincima i programima nego sto bi to bio slucaj bez njihovog koristenja, a također i ukazati na put za siri i pravedniji pristup obrazovanju i zaposljavanju. Međutim, losa upotreba testova može dovesti do zamjetne stete nanesene ispitanicima i drugim sudionicima u procesu donosenja odluka na temelju testovnih podataka. Cilj Standarda je promoviranje kvalitetne i eticne upotrebe testova te uspostavljanje osnovice za ocjenu kvalitete postupaka testiranja. Svrha objavljivanja Standarda je uspostavljanje kriterija za evaluaciju testova, provedbe testiranja i posljedica upotrebe testova. Iako bi evaluacija prikladnosti testa ili njegove primjene trebala ovisiti prvenstveno o strucnim misljenjima, Standardi pružaju okvir koji osigurava obuhvacanje svih relevantnih pitanja. Bilo bi poželjno da svi autori, sponzori, nakladnici i korisnici profesionalnih testova usvoje Standarde te da poticu druge da ih također prihvate.

Standards for educational and psychological testing

https://science.sciencemag.org/content/sci/5/110/206.full.pdf

The american psychological association.

Much has been written in the educational psychology literature about effective feedback and how to deliver it. However, it is equally important to understand how learners actively receive, engage with, and implement feedback. This article reports a systematic review of the research evidence pertaining to this issue. Through an analysis of 195 outputs published between 1985 and early 2014, we identified various factors that have been proposed to influence the likelihood of feedback being used. Furthermore, we identified diverse interventions with the common aim of supporting and promoting learners’ agentic engagement with feedback processes. We outline the various components used in these interventions, and the reports of their successes and limitations. Moreover we propose a novel taxonomy of four recipience processes targeted by these interventions. This review and taxonomy provide a theoretical basis for conceptualizing learners’ responsibility within feedback dialogues, and for guiding the strategic design and evaluation of interventions.

https://www.tandfonline.com/doi/pdf/10.1080/00461520.2016.1207538

Supporting learners’ agentic engagement with feedback:a systematic review and a taxonomy of recipience processes

Intelligent Tutoring Systems (ITS) are computer programs that model learners’ psychological states to provide individualized instruction. They have been developed for diverse subject areas (e.g., algebra, medicine, law, reading) to help learners acquire domain-specific, cognitive and metacognitive knowledge. A meta-analysis was conducted on research that compared the outcomes from students learning from ITS to those learning from non-ITS learning environments. The meta-analysis examined how effect sizes varied with type of ITS, type of comparison treatment received by learners, type of learning outcome, whether knowledge to be learned was procedural or declarative, and other factors. After a search of major bibliographic databases, 107 effect sizes involving 14,321 participants were extracted and analyzed. The use of ITS was associated with greater achievement in comparison with teacher-led, large-group instruction (g .42), non-ITS computer-based instruction (g .57), and textbooks or workbooks (g .35). There was no significant difference between learning from ITS and learning from individualized human tutoring (g –.11) or small-group instruction (g .05). Significant, positive mean effect sizes were found regardless of whether the ITS was used as the principal means of instruction, a supplement to teacher-led instruction, an integral component of teacher-led instruction, or an aid to homework. Significant, positive effect sizes were found at all levels of education, in almost all subject domains evaluated, and whether or not the ITS provided feedback or modeled student misconceptions. The claim that ITS are relatively effective tools for learning is consistent with our analysis of potential publication bias.

/pdf/intelligent-tutoring-systems-and-learning-outcomes-a-meta-1vz988s9i8.pdf

Intelligent tutoring systems and learning outcomes: A meta-analysis

The effects of written feedback in a computer-based assessment for learning on students' learning outcomes were investigated in an experiment at a Higher Education institute in the Netherlands Students were randomly assigned to three groups, and were subjected to an assessment for learning with different kinds of feedback These are immediate knowledge of correct response (KCR) + elaborated feedback (EF), delayed KCR + EF, and delayed knowledge of results (KR) A summative assessment was used as a post-test No significant effect was found for the feedback condition on student achievement on the post-test Results suggest that students paid more attention to immediate than to delayed feedback Furthermore, the time spent reading feedback was positively influenced by students' attitude and motivation Students perceived immediate KCR + EF feedback to be more useful for learning than KR Students also had a more positive attitude towards feedback in a CBA when they received KCR + EF rather than KR only

Effects of feedback in a computer-based assessment for learning

Researchers have shown in multiple studies that simulations and games can be effective and powerful tools for learning and instruction (cf Mitchell & Savill-Smith, 2004; Kirriemuir & McFarlane, 2004) Most of these studies deploy a traditional pretest-posttest design in which students usually do a paper-based test (pretest) then play the simulation or game and subsequently do a second paper-based test (posttest) Pretest-posttest designs treat the game as a black box in which something occurs that influences subsequent performance on the posttest (Buckley, Gobert, Horwitz, & O'Dwyer, 2010) Less research has been done in which game play product data or process data itself are used as indicators of student proficiency in some area However, the last decade researchers have started focusing on what is happening inside the black box to an increasing extent and the literature on the topic is growing To our knowledge, no systematic reviews have been published that investigate the psychometric analysis of performance data of simulation-based assessment (SBA) and game-based assessment (GBA) Therefore, in Part I of this article, a systematic review on the psychometric analysis of the performance data of SBA is presented The main question addressed in this review is: 'What psychometric strategies or models for treating and analyzing performance data from simulations and games are documented in scientific literature?' Then, in Part II of this article, the findings of our review are further illustrated by presenting an empirical example of the - according to our review - most applied psychometric model for the analysis of the performance data of SBA, which is the Bayesian network Both the results from Part I and Part II assist future research into the use of simulations and games as assessment instruments We present a review on psychometric analysis of simulation-based assessmentWe performed the review from the Evidence-Centered Design framework perspectiveThe Bayes Net is the most used psychometric model for simulation-based assessmentWe present an example of a Bayes Net of a real simulation-based assessmentWe make recommendations regarding the development of a simulation-based assessment

Psychometric analysis of the performance data of simulation-based assessment

Computerized adaptive tests (CATs) are individualized tests that, from a measurement point of view, are optimal for each individual, possibly under some practical conditions. In the present study, ...

Optimal Testing With Easy or Difficult Items in Computerized Adaptive Testing

Validity is the most important quality aspect of tests and assessments, but it is not clear how validity can be evaluated This article presents a procedure for the evaluation of validity and validation which is an extension of the argument-based approach to validation The evaluation consists of three criteria to evaluate the interpretive argument, the validity evidence provided, and the validity argument This procedure is illustrated with an existing assessment: the driver performance assessment The article concludes with recommendations for the application of the procedure Keywords Competence assessment, validity, validation, argument-based approach, evaluation Expressive Reading Aloud on IEA-PIRLS 2006 Gabriella Agrusti This in-depth research study was aimed at investigating the effects of readaloud modification on students’ performance on PIRLS 2006 reading comprehension tests, in two different forms: expressive reading and neutral reading In Italy international comparative surveys often represent the main reference measure for student achievements in basic skills, but few experimental designs descending from secondary analyses attempt to investigate possible relationships among variables in order to translate results in suggestions for teaching practices The present study was intended as a first step in this direction, analyzing if specific aspects of reading aloud can influence students’ achievements in comprehension Differences in means between groups were found statistically significant between expressive reading aloud administration and silent standard administration, though the strong concurrent validity of PIRLS 2006 test items allowed only a small variance in results due to experimental modification Neutral reading aloud alone did not have significantly different effects on test results, confirming the general assumption that at this age word recognition skill is at an advanced level More evident results were shown in items focused on the processes of interpreting and integrating ideas and information, highlighting how inferential more than retrieval processes are influenced by reading aloud, that efficiently convey the gist of the story and some of the implications in meaning of complex semantic topics, such as the identification and description of main characters’ feelings

Evaluation of Validity and Validation by Means of the Argument-based Approach

https://ris.utwente.nl/ws/files/6568695/1-s2.0-S0191491X16300852-main.pdf

Theo J. H. M. Eggen

Papers

Effects of feedback in a computer-based assessment for learning

Psychometric analysis of the performance data of simulation-based assessment

Optimal Testing With Easy or Difficult Items in Computerized Adaptive Testing

Evaluation of Validity and Validation by Means of the Argument-based Approach

Formative use of test results: A user’s perspective