Showing papers in "Assessment & Evaluation in Higher Education in 1996"

The Importance of Marking Criteria in the Use of Peer Assessment

Abstract: This paper reports a study which implemented and evaluated a method of peer assessment, as a formative and summative assessment procedure. Pairs of first‐year undergraduate biology students were asked to complete a poster assignment on a specific aspect of nerve physiology. This paper contains details of a method which allows student peer and tutor marking of work against the individual marking criteria to be evaluated. The results show that a comparison between the tutor and the student peer mark may be misleading as a guide to the validity of peer assessment. The importance of considering the individual sections of the marking criteria is illustrated. It was found that when the individual criteria were analysed the number of students marking the same as the tutor ranged from 31% to 62%. It also became clear that specific areas of the marking criteria were prone to over and under‐marking. Analysis of student feedback forms showed that students not only liked carrying out peer assessment, but fel...

Assessing Learning Quality: reconciling institutional, staff and educational demands

Abstract: Two major frameworks for educational decision making, including decisions on assessment, can be distinguished: quantitative, whichis adequate for construing some kinds of learning; and qualitative, which should be the appropriate framework for enabling decisions flowing from most tertiary educational aims. However, for various reasons, institutions implicitly encourage a quantitative framework for assessment‐related decision making, particularly evident in the recent emphasis on accountability and performance indicators. This is unfortunate because, through the backwash effect, quantitative modes of assessment encourage surface approaches to learning, which typically lead to low cognitive‐level outcomes that are not compatible with stated course objectives. It is argued that an institution and its educational practices comprise a system in equilibrium, and that if educational goals are to be realised, the whole system needs to be compatible with those aims.

A Survey of Methods of Deriving Individual Grades from Group Assessments

Abstract: This paper surveys the different methods of assessing groups of students which have been reported in the literature and describes some novel methods which have been used at the University of Sunderland. It relates the work of Schechtman to peer assessment in a group setting, discusses some observations on self‐assessment and raises some questions for debate and future research.

Reflection: can we assess it? Should we assess it?

Abstract: This paper reports on a study undertaken by two university‐based teacher educators into an aspect of professional practice. The feasibility and desirability of assessing reflection demonstrated by student teachers studying early childhood literacy was investigated. While reaffirming the importance of developing reflective practitioners, the study highlighted the difficulties of equitably or meaningfully assessing reflection. Use of alternative methodologies was called for in future research into reflection

Tutor Versus Peer Group Assessment of Student Performance in a Simulation Training Exercise

Abstract: The performance of a group of third‐year higher diploma students from the Department of Hotel & Tourism in a simulation training exercise was assessed separately by the tutor and peer group, using an identical checklist. Ninety‐six pairs of tutor and peer group assessment marks were obtained and compared. Results showed that there was some degree of agreement between tutor and peer group markings, although the level of congruence was somewhat less than reported in earlier studies. However, when the tutor and peer group average marks were converted to grades, agreement was found in only 46.9% of the cases and the level of agreement between the two gradings was not statistically significant. Significant differences were also found in the variability of tutor and peer group markings. Furthermore, there were indications that peer group markings tended to be more stringent than tutor markings in the higher mark range, but relatively more lenient in the lower mark range. This trend was particularly obv...

The Nominal Group Technique: extending the evaluation of students' teaching and learning experiences

Abstract: Obtaining and evaluating student opinion about their teaching and learning experiences can, if taken seriously, be a complex process. There are not many approaches available for student evaluations, and there has in our view, been an over dependence on course evaluation questionnaires. The Nominal Group Technique (NGT) is a less widely used approach, which this paper argues provides a potential way of extending the evaluation of students' experiences of courses. The paper specifically considers the use of the NGT in the context of an evaluation of the first three cohorts of students on a new degree course. The strengths and weaknesses of the approach as they have emerged in this evaluation are considered. Attention is also paid to evidence concerning the use of the NGT in other contexts. The paper concludes that the NGT has much to offer within the portfolio of student evaluation methods currently being used in higher education. Its particular strengths are assessed in relation to certain possibl...

Two Models of Educational Assessment.

Abstract: Many educational developments in recent decades pose a serious challenge to the traditional scientific measurement model that has dominated assessment practices. The scientific measurement model has led to an over‐emphasis on statistical tests and the reification of single measure test scores. The educational developments that challenge the scientific measurement model include problem‐based learning, newer understandings of cognition, and the rise of performance assessment. These developments reflect widespread attempts by educators to reform assessment practices so as to encourage more effective learning. As a result, a new model of educational assessment, which we call the judgemental model, is emerging. The basic assumptions, features and appropriate uses of these two assessment models are compared and contrasted by referring them to a three‐level conceptual model of education, training and assessment for workplace performance.

Peer Assessment of Group Projects in Civil Engineering

Abstract: This article traces one university's approaches of peer assessment to develop fair and reliable systems of mark distribution resulting from group projects and to adapt one model to the specific requirements of civil engineering. It is the result of a two and a half years' experience with peer assessment in classes of different combinations of size, entry qualifications and experience. Initially, the methods devised by Goldfinch & Raeside (1990) were used; the relevance and drawbacks of this two‐part method of assessment in the field of civil engineering are discussed and new ideas are proposed. That article had suggested that the model is transferable. It has proved to be the case and in this discipline, that project diaries which the tutor uses to allocate individuals a mark, offer a unique alternative to Part 1 of the assessment process.

Quality Assurance: international trends, preoccupations and features

Abstract: Many countries are grappling with similar issues in higher education, the main issues being mass higher education and its many consequences, such as the cost, who should pay, appropriate types of education, appropriate types of institution, how to evaluate the education, and who should evaluate it Central to these considerations is the assurance of the quality of institutional activities, namely research and other creative activities, teaching in all its forms, and consulting and general service to the local and wider communities Currently, power balances are converging within each educational system, and becoming more similar across systems Consequently, similar methods for quality assurance (QA) are being adopted This paper describes some of the commonly occurring preoccupations with quality, and some of the procedures and tools being used to address them

Assessing the Practicum

Abstract: The practicum appears in many different forms in professional education — as field placement, cooperative education, sandwich programs, internship and clinical placement. This review of current literature reveals many different conceptions of the kinds of learning outcomes that can be achieved through learning in the workplace and the contribution that the practicum can make to professional education. In consequence, assessment philosophies and methods are equally varied. Five distinct approaches to assessment of the practicum are identified and the strengths and weaknesses of each are evaluated. While assessment of the practicum in some programs has been very limited, in the best examples cited, integration of the practicum with the educational program has provided opportunities for students to bring together a range of knowledge and skills in a complex natural environment. Well‐designed assessment requirements can make a significant contribution to learning in the practicum by putting pressure ...

Evaluating Undergraduate Education: the use of broad indicators

Abstract: The aim of this study was to examine the extent to which broad indicators of performance, such as student satisfaction with program, teaching, student life and experiences after graduation, could be used for program improvement. Information was obtained from graduates of a major research university in Canada who had received an undergraduate professional degree during the period from 1983 to 1992. Perceived quality of teaching was found to contribute significantly to graduates' rating of the overall quality of their academic program. The most frequently mentioned meaningful feature of their undergraduate education for alumni was the development of the ability to think. The results suggest the need for policy changes to improve teaching and the need for greater organizational coherence in post‐secondary institutional evaluations.

Student Perceptions of Assessment Methods

Abstract: With the introduction of teaching quality audits, attention is being directed towards methods of assessment. In the present study students were asked what qualities they perceived in continuous assessment and examinations. This was considered important in light of the view that assessment techniques can be used to motivate students. It appeared that in many respects students regarded the former as being fairer, and measuring a greater range of abilities. The main findings were that students' reactions varied according to age and gender, but that overall the view was that continuous assessment should not be involved in much more than half of their grade measurement.

A Peer‐tutoring Scheme to Support Independent Learning and Group Project Work in Mathematics

Abstract: A peer‐tutoring scheme was introduced to an undergraduate mathematics module. This was to provide a learning support for an independent learning programme. These student support groups also functioned as task groups for group project work. In the independent learning programme, students were directed to read selected passages of text, to attempt certain exercises and to devise peer assessment tasks. For some of the sessions senior students were present and functioned as additional peer tutors. To assess the students’ attitudes to the peer‐supported independent learning programme, an Attitudes to Peer‐tutoring Questionnaire was constructed. The results show that the students readily accepted the need to work in groups and to support one another. Overall, 78% felt that they could work easily without pressure and that the sessions were not a complete waste of time. However, 65% of the students did not appear to enjoy the independent learning sessions and felt that they preferred to be responsible on...

The Reliability, Validity, and Evaluation of the Objective Structured Clinical Examination in Podiatry (Chiropody).

Abstract: The objective structured clinical examination (OSCE), initially developed for use in undergraduate medical training, has been adapted for use for the assessment of clinical skills in podiatry (chiropody) students. The introduction of this system of examination was accompanied by a 12‐month evaluation programme where issues of validity, repeatability, and stability of student performance over time were considered. The evaluation programme revealed relatively low levels of repeatability, high levels of construct and criterion validity, and good stability, of performance over time. Participating students were also surveyed for their level of acceptance of this format of clinical examination. Students demonstrated an overwhelming acceptance and support of the OSCE.

Physiotherapy Student Clinical Education: the influence of subjective judgements on observational assessment

Abstract: The process of assessing students on clinical placements has to date been under‐researched. In particular, while the involvement of subjective judgements in assessing students has been discussed in the literature, there has been no attempt to look for systematic evidence. This paper comprises the first attempt to relate judgement theory to the assessment of vocational education. Preliminary results demonstrate that physiotherapy assessors made subjective judgements of their students, that these contributed to the determination of assessment grades, and that in certain cases these judgements were erroneous. The relationship of these subjective judgements to the awarding of grades via a supposedly objective assessment instrument is not yet known. Given the widespread use of such assessment instruments across different professions, the question raised is whether all observational assessment systems suffer from similar problems. If students are being judged on criteria not included on assessment inst...

Rating Teaching in the USA: probing the qualifications of student raters and novice teachers

Abstract: In the USA, student ratings of their instructors are routinely used by administrators in higher education in making decisions regarding instructors' salary adjustments, tenure and promotion. However, when the rating qualifications of amateur student raters and novice public school teachers who have received training that should have enabled them to become qualified raters are examined closely, there are good reasons for believing that both groups of raters are not qualified to give reliable ratings on most high‐inference questionnaire items.

An Analysis of Degree Results in Geography by Gender

Abstract: Chi‐square analysis is used to test for differences in the degree results of male and female geography graduates from 42 UK universities between 1973 and 1993. The combined results of these universities (i.e. departments at subject‐specific level) show a persistent and striking overrepresentation of males with thirds and pass degrees together with a less obvious, but statistically significant, overrepresentation of males achieving firsts. Corresponding analyses for individual departments emphasise the need for caution in deriving generalisations from aggregate data. A significant difference (at the 99% level) in the cumulative annual degree results of males and females is found in 10 of the 42 departments. The sources of these differences vary and it is suggested that a distinction is drawn between general factors responsible for the pervasive overrepresentation of males in the third/pass categories and department‐specific factors responsible for gender differences in other degree classes, especi...

The Introduction of Computer-Based Testing on an Engineering Technology Course.

Abstract: The current situation in higher education is one of increasing student numbers with a decreasing unit of resource. To help cope with this situation lecturers are exploring the possibility of using non‐traditional methods in some aspects of their work. One possibility is to apply new technology in the assessment of students. This paper presents a controlled comparison between traditional paper‐based tests and those using a computer. It is concluded that the new technique is acceptable to students and produces results with no deterioration in their validity; it also has great potential for the saving of staff time.

An Analysis of Student Grade Expectations and Marker Consistency

Abstract: This study sets out to assess the extent to which final‐year students following a four year degree in education have a realistic sense of the strengths and weaknesses of an independent research study they are required to undertake. An analysis of the self‐assessment grades and those of their tutors reveals a poor match, whilst the criteria students use to assess their studies emphasise lower order criteria such as style and presentation, and largely ignore higher order criteria concerned with theoretical and conceptual understanding and the quality of discussion. The paper considers why students appear not to have theorised their experiences, and why there is only limited evidence of a critical understanding of academic writing.

Use of Instructional Feedback and Modification Methods among University Faculty

Abstract: This study focused on methods used by faculty to gather feedback and how this feedback was used to modify instruction. Data were gathered to determine the degree to which these methods of feedback and modification of instruction were dependent upon variables such as class size, teaching experience, pedagogical training, academic rank and status, gender and academic discipline. It was found that the amount of feedback and modification of instruction did not vary significantly across the variables of teaching experience and class size. The analyses examining the effects of the remaining independent variables (i.e. college of study, pedagogical training, gender, rank, status) did detect statistically significant results. Specifically, those instructors who had received some training in pedagogical methods gathered feedback about their teaching more than those instructors without such pedagogical training. Also, faculty from the Colleges of Education and Liberal Arts used more methods to gather feedb...

Simulations for the Assessment of Counselling Skills

Abstract: Assessment of students’ communication skills after a course in counselling, in which the students are prepared for work practice, requires another assessment method than the traditional written examination. In this paper we describe the construction and evaluation of simulations. The results of an investigation into the quality of the simulations show that they enable objective assessment of students’ mastery of skills in the situation which is simulated. In order to achieve generalisable scores, i.e. scores which indicate how well a student will perform in counselling interviews in general, more than one simulation has to be administered. Although the simulation is not an efficient instrument, its benefits are high: weaknesses in students’ performances will be detected so that remedial teaching can be offered, and clinical practice can be preserved for counsellors with a less than sufficient ability in counselling skills

Cultural Issues in Evaluation Studies of Middle Eastern Higher Education

Abstract: Middle Eastern candidates for higher degrees often set out with proposals which include a significant element of evaluation of aspects of higher education. These may raise cultural issues. Fourth generation evaluation procedures take the cultural context seriously. This is likely to throw up potentially important issues for candidates and their supervisors. This paper illustrates these matters in the current context of Middle Eastern higher education. It discusses the role of evaluation studies in this field, together with issues which candidates and their supervisors may confront when carrying out such research studies there. National cultures may be approached as sites of cultural contestation. Some forms of evaluation research have a role to play in this

Key Teaching Roles of a University Lecturer and their Integration into the Quality Systems of a New Zealand University

Abstract: This paper describes the results of a training needs analysis of the position of university lecturer at Victoria University of Wellington. The analysis identified 10 key roles which together encompass the range of teaching activities expected of lecturers. The analysis also identified an associated academic staff development programme to guide the work of the University Teaching Development Centre. Because of the growing acceptance of the results of the analysis, the University has also integrated the findings into its quality management procedures. The paper describes the use of the information in promotion and appraisal procedures, and the links between the key roles and other quality assurance mechanisms. The paper concludes with a brief discussion of the gaps in the University's quality assurance systems as they relate to the key roles

Effective Student Focus Groups: the bright and early approach

Abstract: The perceptions of enrolled students are an important part of academic programme review. However, identification of appropriate students and scheduling of suitable times can be difficult. By obtaining faculty assistance in selection of students and scheduling focus groups as breakfast meetings, a high level of attendance and participation was achieved.

Undergraduate Degree Awards in Teacher Education: constant standards in a changing context

Abstract: In the past, assessment in higher education has been treated as a matter determined by, and confidential to, Boards of Examiners. In future, however, this may have to change in response to a number of changes in the context of higher education. The student population is more diverse and is taking an increasing control of the curriculum through the modularisation of courses. Student charters instil expectations that details of assessment procedures will be made explicit and published. Awards may increasingly be challenged. This paper examines a number of issues which will have to be debated more openly in the future. They include the statistics of aggregating component grades into degree classes, the widening range of skills covered by higher education courses and the most appropriate formats for summative reporting. The paper aims not to offer answers but to help to clarify some of the questions around which debate will be required.

An Investigation of the Effectiveness of an Evaluation Instrument Designed for Use with Modular Masters Courses in a Range of International Contexts

Abstract: In a previous paper the construction of an instrument designed to be of use in the evaluation of modular Masters courses for teachers and administrators in international schools was described. Initial impressions of the efficacy of the instrument were encouraging, and its use was extended in a number of ways within the scheme of continuing professional development organised through the Centre for the study of Education in an International Context at the University of Bath. In particular, the evaluation model was extended to three different modules taught in an increased number of Study Centre locations, by different tutors, to groups of different sizes and composition. Under such a range of conditions the question arises of parity of quality in the delivery of the modules, and the usefulness of the evaluation instrument in contributing to an assessment of the quality of teaching and learning was explored. This paper describes the results of an investigation undertaken to test the robustness of th...

A Testing Time: The Role of Subjective Practices in Making Sense of Student Performance.

Abstract: This paper examines some of the practices and accompanying assumptions that underlie the work of assessing students' written performance. Examiners are shown to give meaning to, and to construct and reconstruct, the answers that their students provide. In doing this resort is made to common sense or taken for granted concepts, many of which involve labelling. Political processes such as power, negotiation and compromise are also identified. In a climate of moves towards mass higher education where written and distant assessment is likely to grow at the expense of face‐to‐face contact, the tendency of examiners, highlighted in this paper, to rely on interpretive and negotiated methods is not only inevitable but justified.

A Case Study of Student Teachers' Science Placements.

Abstract: The role of placements in the development of the education/industry partnership is considered with particular reference to initial teacher education. A detailed study of a placement scheme run by an ITE institution and Teacher Placement Organisers (TPO) is described and analysed to highlight issues regarding the nature and process of students' learning from placements. In the context of science and technology it is suggested that modifications to the content of subjects are necessary if students are successfully to investigate the workplace using their subject understanding. However, there are dangers that learning might become narrowly vocational and it is argued that in the evaluation of learning it is necessary to consider the use of inquiry skills in relation to the subject‐specific context if educational concerns are to be safe‐guarded.