Showing papers by "Frank Goldhammer published in 2020"

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects.

Abstract: International large-scale assessments, such as the Programme for International Student Assessment (PISA), are conducted to provide information on the effectiveness of education systems. In PISA, the target population of 15-year-old students is assessed every three years. Trends show whether competencies have changed in the countries between PISA cycles. In order to provide valid trend estimates, it is desirable to retain the same test conditions and statistical methods in all PISA cycles. In PISA 2015, however, the test mode changed from paper-based to computer-based tests, and the scaling method was changed. In this paper, we investigate the effects of these changes on trend estimation in PISA using German data from all PISA cycles (2000 to 2015). Our findings suggest that the change from paper-based to computer-based tests could have a severe impact on trend estimation but that the change of the scaling model did not substantially change the trend estimates.

ICT Engagement: a new construct and its assessment in PISA 2015

Abstract: As a relevant cognitive-motivational aspect of ICT literacy, a new construct ICT Engagement is theoretically based on self-determination theory and involves the factors ICT interest, Perceived ICT competence, Perceived autonomy related to ICT use, and ICT as a topic in social interaction. In this manuscript, we present different sources of validity supporting the construct interpretation of test scores in the ICT Engagement scale, which was used in PISA 2015. Specifically, we investigated the internal structure by dimensional analyses and investigated the relation of ICT Engagement aspects to other variables. The analyses are based on public data from PISA 2015 main study from Switzerland (n = 5860) and Germany (n = 6504). First, we could confirm the four-dimensional structure of ICT Engagement for the Swiss sample using a structural equation modelling approach. Second, ICT Engagement scales explained the highest amount of variance in ICT Use for Entertainment, followed by Practical use. Third, we found significantly lower values for girls in all ICT Engagement scales except ICT Interest. Fourth, we found a small negative correlation between the scores in the subscale “ICT as a topic in social interaction” and reading performance in PISA 2015. We could replicate most results for the German sample. Overall, the obtained results support the construct interpretation of the four ICT Engagement subscales.

More Than (Single) Text Comprehension? - On University Students' Understanding of Multiple Documents.

Abstract: The digital revolution has made a multitude of text documents from highly diverse perspectives on almost any topic easily available. Accordingly, the ability to integrate and evaluate information from different sources, known as multiple document comprehension, has become increasingly important. Because multiple document comprehension requires the integration of content and source information across texts, it is assumed to exceed the demands of single text comprehension due to the inclusion of two additional mental representations: the integrated situation model and the intertext model. To date, there is little empirical evidence on commonalities and differences between single text and multiple document comprehension. Although the relationships between single text and multiple document comprehension can be well distinguished conceptually, there is a lack of empirical studies supporting these assumptions. Therefore, we investigated the dimensional structure of single text and multiple document comprehension with similar test setups. We examined commonalities and differences between the two forms of text comprehension in terms of their relations to final school exam grades, level of university studies and university performance. Using a sample of n = 501 students from two German universities, we jointly modeled single text and multiple document comprehension and applied a series of regression models. Concerning the relationship between single text and multiple document comprehension, confirmatory dimensionality analyses revealed the best fit for a model with two separate factors (latent correlation: 0.84) compared to a two-dimensional model with cross-loadings and fixed covariance between the latent factors and a model with a general factor. Accordingly, the results indicate that single text and multiple document comprehension are separable yet correlated constructs. Furthermore, we found that final school exam grades, level of university studies and prior university performance statistically significant predicted both single text and multiple document comprehension and that expected future university performance was predicted by multiple document comprehension. There were also statistically significant relationships between multiple document comprehension and these variables when single text comprehension was taken into account. The results imply that multiple document comprehension is a construct that is closely related to single text comprehension yet empirically differs from it.

Learning Analytics and eAssessment : Towards Computational Psychometrics by Combining Psychometrics with Learning Analytics

Abstract: From a psychometric point of view, assessment means to infer what a learner knows and can do in the real world from limited evidence observed in a standardized testing situation. From a learning analytics perspective assessment means to observe real behavior in digital learning environments to conclude the learner status with the intent to positively influence the learning process. Although psychometrics and learning analytics share similar goals, for instance, formative assessment, while applying different methods and theories, the two disciplines are so far highly separated. This chapter aims at paving the way for an advanced understanding of assessment by comparing and integrating the learning analytics and the psychometric approach of assessment. We will discuss means to show this new way of assessment of educational concepts such as (meta-) cognition, motivation, and reading comprehension skills that can be addressed either from data-driven approach (learning analytics) or from a theory-driven approach (psychometrics). Finally, we show that radically new ways of assessment are located in the middle space where both disciplines are combined into a new research discipline called ‘Computational Psychometrics’.

Entwicklung und Skalierung eines Tests zur Erfassung des Verständnisses multipler Dokumente von Studierenden

Abstract: Zusammenfassung. Das Verstandnis multipler Dokumente (Multiple Document Comprehension, MDC) wird als Fahigkeit verstanden, aus verschiedenen Informationsquellen eine integrierte Reprasentation eine...

Evaluation of Online Information in University Students: Development and Scaling of the Screening Instrument EVON.

Abstract: As Internet sources provide information of varying quality, it is an indispensable prerequisite skill to evaluate the relevance and credibility of online information. Based on the assumption that competent individuals can use different properties of information to assess its relevance and credibility, we developed the EVON (evaluation of online information), an interactive computer-based test for university students. The developed instrument consists of eight items that assess the skill to evaluate online information in six languages. Within a simulated search engine environment, students are requested to select the most relevant and credible link for a respective task. To evaluate the developed instrument, we conducted two studies: (1) a pre-study for quality assurance and observing the response process (cognitive interviews of n = 8 students) and (2) a main study aimed at investigating the psychometric properties of the EVON and its relation to other variables (n = 152 students). The results of the pre-study provided first evidence for a theoretically sound test construction with regard to students' item processing behavior. The results of the main study showed acceptable psychometric outcomes for a standardized screening instrument with a small number of items. The item design criteria affected the item difficulty as intended, and students' choice to visit a website had an impact on their task success. Furthermore, the probability of task success was positively predicted by general cognitive performance and reading skill. Although the results uncovered a few weaknesses (e.g., a lack of difficult items), and the efforts of validating the interpretation of EVON outcomes still need to be continued, the overall results speak in favor of a successful test construction and provide first indication that the EVON assesses students' skill in evaluating online information in search engine environments.

Multiple Document Comprehension of University Students

Abstract: Multiple document comprehension is the ability to construct an integrated representation of a specific topic based on several sources. It is an important competence for university students; however, there has been so far no established instrument to assess multiple document comprehension in a standardized way. Therefore, we developed a test covering four theory-based cognitive requirements: The corroboration of information across texts, the integration of information across texts, the comparison of sources and source evaluations across texts, and the comparison of source-content links across texts. The developed test comprised 174 items and was empirically examined in a study with 310 university students. Several items had to be excluded due to psychometric misfit and differential item functioning. The resulting final test contains 67 items within 5 units (i.e., test structures of 2–3 texts and related items) and has been shown to fit a unidimensional IRT Rasch model. The test score showed expected relationships to the final school exam grade, the study level (Bachelor/Master), essay performance, sourcing behavior, as well as mental load and mental effort.

Using process data to explain group differences in complex problem solving

Abstract: In large-scale assessments, performance differences across different groups are regularly found. These group differences (e.g., gender differences) are often relevant for educational policy decisions and measures. However, the formation of these group differences usually remains unclear. We propose an approach for investigating this formation by considering behavioral process measures as mediating variables between group membership and performance on the 2012 Programme for International Student Assessment complex problem solving (CPS) items. We found that across all investigated countries interactive behavior can fully explain gender differences in CPS, but cannot explain differences between students with and without a migration background. However, in some countries these results differ from the cross-country results. Our results indicate that process measures derived from log data are useful for further investigating and explaining performance differences between girls and boys and students with and without migration background. (PsycInfo Database Record (c) 2020 APA, all rights reserved)

Analysing Log File Data from PIAAC

Abstract: The OECD Programme for the International Assessment of Adult Competencies (PIAAC) was the first computer-based large-scale assessment to provide anonymised log file data from the cognitive assessment together with extensive online documentation and a data analysis support tool. The goal of the chapter is to familiarise researchers with how to access, understand, and analyse PIAAC log file data for their research purposes. After providing some conceptual background on the multiple uses of log file data and how to infer states of information processing from log file data, previous research using PIAAC log file data is reviewed. Then, the accessibility, structure, and documentation of the PIAAC log file data are described in detail, as well as how to use the PIAAC LogDataAnalyzer to extract predefined process indicators and how to create new process indicators based on the raw log data export.

Convergent Evidence for the Validity of a Performance-Based ICT Skills Test

Abstract: . The goal of this study was to investigate sources of evidence of convergent validity supporting the construct interpretation of scores on a simulation-based information and communication ...

Testwertinterpretation, Testnormen und Testeichung

Abstract: In diesem Kapitel werden verschiedene Moglichkeiten beschrieben, wie ein Testergebnis bzw. Testwert deskriptivstatistisch interpretiert werden kann. Bei der normorientierten Interpretation wird der Testwert in einen Normwert transformiert, der einen Vergleich mit den Testwerten anderer Personen einer Bezugsgruppe (den „Testnormen“) erlaubt. Die Testnormen werden im Wege der Testeichung an einer reprasentativen Eichstichprobe gewonnen. Sofern genauere theoretische Vorstellungen bestehen, kann das Testergebnis auch mit einem inhaltlich definierten Kriterium in Bezug gesetzt werden. Dieser Vorgang wird als kriteriumsorientierte Interpretation bezeichnet. Beide Interpretationsarten konnen auch miteinander verbunden werden.

PISA reading. Mode effects unveiled in short text responses

Abstract: Psychological test and assessment modeling 62 (2020) 1, S 85-105 Padagogische Teildisziplin: Empirische Bildungsforschung;

Digitalisierung an Schulen – eine Bestandsaufnahme

Abstract: Frankfurt am Main : DIPF | Leibniz-Institut fur Bildungsforschung und Bildungsinformation 2020, 99 S. Padagogische Teildisziplin: Empirische Bildungsforschung; Padagogische Psychologie;