Flaws in the design, conduct, analysis, and reporting of randomised trials can cause the effect of an intervention to be underestimated or overestimated. The Cochrane Collaboration’s tool for assessing risk of bias aims to make the process clearer and more accurate

https://www.bmj.com/content/bmj/343/bmj.d5928.full.pdf

The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials

Pedagosko i psiholosko testiranje i procjenjivanje spadaju među najvažnije doprinose znanosti o ponasanju nasem drustvu i pružaju temeljna i znacajna poboljsanja u odnosu na ranije postupke. Iako se ne može ustvrditi da su svi testovi dovoljno usavrseni niti da su sva testiranja razborita i korisna, postoji velika kolicina informacija koje ukazuju na ucinkovitost kvalitetnih instrumenata u onim situacijama u kojima je njihovo koristenje potkrijepljeno validacijskim podacima. Pravilna upotreba testova može dovesti do boljih odluka o pojedincima i programima nego sto bi to bio slucaj bez njihovog koristenja, a također i ukazati na put za siri i pravedniji pristup obrazovanju i zaposljavanju. Međutim, losa upotreba testova može dovesti do zamjetne stete nanesene ispitanicima i drugim sudionicima u procesu donosenja odluka na temelju testovnih podataka. Cilj Standarda je promoviranje kvalitetne i eticne upotrebe testova te uspostavljanje osnovice za ocjenu kvalitete postupaka testiranja. Svrha objavljivanja Standarda je uspostavljanje kriterija za evaluaciju testova, provedbe testiranja i posljedica upotrebe testova. Iako bi evaluacija prikladnosti testa ili njegove primjene trebala ovisiti prvenstveno o strucnim misljenjima, Standardi pružaju okvir koji osigurava obuhvacanje svih relevantnih pitanja. Bilo bi poželjno da svi autori, sponzori, nakladnici i korisnici profesionalnih testova usvoje Standarde te da poticu druge da ih također prihvate.

Standards for educational and psychological testing

INTRODUCTION: Appraising the quality of studies included in systematic reviews combining qualitative and quantitative evidence is challenging. To address this challenge, a critical appraisal tool was developed: the Mixed Methods Appraisal Tool (MMAT). The aim of this paper is to present the enhancements made to the MMAT. DEVELOPMENT: The MMAT was initially developed in 2006 based on a literature review on systematic reviews combining qualitative and quantitative evidence. It was subject to pilot and interrater reliability testing. A revised version of the MMAT was developed in 2018 based on the results from usefulness testing, a literature review on critical appraisal tools and a modified e-Delphi study with methodological experts to identify core criteria. TOOL DESCRIPTION: The MMAT assesses the quality of qualitative, quantitative, and mixed methods studies. It focuses on methodological criteria and includes five core quality criteria for each of the following five categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, and (e) mixed methods. CONCLUSION: The MMAT is a unique tool that can be used to appraise the quality of different study designs. Also, by limiting to core criteria, the MMAT can provide a more efficient appraisal.

/pdf/the-mixed-methods-appraisal-tool-mmat-version-2018-for-pv84g3e3lr.pdf

The Mixed Methods Appraisal Tool (MMAT) Version 2018 for Information Professionals and Researchers.

Objectives: The aim of this study was to develop a critical appraisal (CA) tool that addressed study design and reporting quality as well as the risk of bias in cross-sectional studies (CSSs). In addition, the aim was to produce a help document to guide the non-expert user through the tool.

Design: An initial scoping review of the published literature and key epidemiological texts was undertaken prior to the formation of a Delphi panel to establish key components for a CA tool for CSSs. A consensus of 80% was required from the Delphi panel for any component to be included in the final tool.

Results: An initial list of 39 components was identified through examination of existing resources. An international Delphi panel of 18 medical and veterinary experts was established. After 3 rounds of the Delphi process, the Appraisal tool for Cross-Sectional Studies (AXIS tool) was developed by consensus and consisted of 20 components. A detailed explanatory document was also developed with the tool, giving expanded explanation of each question and providing simple interpretations and examples of the epidemiological concepts being examined in each question to aid non-expert users.

Conclusions: CA of the literature is a vital step in evidence synthesis and therefore evidence-based decision-making in a number of different disciplines. The AXIS tool is therefore unique and was developed in a way that it can be used across disciplines to aid the inclusion of CSSs in systematic reviews, guidelines and clinical decision-making.

https://bmjopen.bmj.com/content/bmjopen/6/12/e011458.full.pdf

Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS)

Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review

A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed

A general critical appraisal tool: An evaluation of construct validity

Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs

Objective: To examine the reliability of scores obtained from a proposed critical appraisal tool (CAT). Study Design and Setting: Based on a random sample of 24 health-related research papers, the scores from the proposed CAT were examined using intraclass correlation coefficients (ICCs), generalizability theory, and participants’ feedback. Results: The ICC for all research papers was 0.83 (consistency) and 0.74 (absolute agreement) for four participants. For individual research designs, the highest ICC (consistency) was for qualitative research (0.91) and the lowest was for descriptive, exploratory and observational research (0.64). The G study showed a moderate research design effect (32%) for scores averaged across all papers. The research design effect was mainly in the Sampling, Results, and Discussion categories (44%, 36%, and 34%, respectively). The scores for research

ORIGINAL ARTICLES Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs

In systematic reviews, evidence-based practice and journal clubs critical appraisal tools are used to rate research papers. However, little evidence exists on whether the critical appraisal tool, subject matter knowledge or research design knowledge affect the appraisal of research papers. A match paired randomised trial was conducted in August/September 2010 in the Faculty of Medicine, Health and Molecular Science, James Cook University, Australia. Ten participants in total were randomly assigned to two groups using either an informal appraisal of research (IA group) or the Crowe Critical Appraisal Tool (CCAT group), a general critical appraisal tool. Participant independently appraised five research papers, where each paper had a different research design. The scores allocated to the papers by each group were analysed. The intraclass correlation coefficient for absolute agreement was 0.76 for the informal appraisal group and 0.88 for the Crowe Critical Appraisal Tool group. The G study showed that in the informal appraisal group 24% of variance in scores was attributable to either the rater or paper × rater interactions, whereas this was 12% in the Crowe Critical Appraisal Tool group. Analysis of covariance showed that there were statistically significant results in the informal appraisal group for subject matter knowledge (F(1,18) = 7.03, P < 0.05 1 tailed, partial η² = 0.28) and rater (F(4,18) = 4.57, P < 0.05 1 tailed, partial η² = 0.50). Kendall's tau correlation coefficient also showed a significant weak positive relationship (τ = 0.38, P = 0.03) between total score and subject matter knowledge for the informal appraisal group. The Crowe Critical Appraisal Tool was more reliable than an informal appraisal of the research papers. In the informal appraisal group, there were significant effects for rater and subject matter knowledge, whereas the Crowe Critical Appraisal Tool almost eliminated the rater effect, and no subject matter knowledge effect was apparent. There was no research design knowledge effect in either group. The Crowe Critical Appraisal Tool provided much better score reliability and should help readers with different levels and types of knowledge to reach similar conclusions about a research paper.

Michael Crowe

Papers

A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed

A general critical appraisal tool: An evaluation of construct validity

Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs

ORIGINAL ARTICLES Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs

Comparison of the effects of using the Crowe Critical Appraisal Tool versus informal appraisal in assessing health research: a randomised trial.