Pythia reloaded: an intelligent unit testing-based code grader for education
14 Jul 2015-pp 5-8
TL;DR: A platform combining the advantages of unit testing and competition graders to provide a unit testing-based grader that assesses codes and produces relevant and "intelligent" feedbacks to support learning is proposed.
Abstract: Automatic assessment of code to support education is an important feature of many programming learning platforms. Unit testing frameworks can be used to perform a systematic functional test of codes; they are mainly used by developers. Competition graders can be used to safely execute code in sandboxed environments; they are mainly used for programming contests. This paper proposes a platform combining the advantages of unit testing and competition graders to provide a unit testing-based grader. The proposed platform assesses codes and produces relevant and "intelligent" feedbacks to support learning. The paper presents the architecture of the platform and how the unit tests are designed.
Citations
More filters
TL;DR: The development of an application for the grading and provision of feedback on educational processes, named EduZinc, enables instructors to go through the complete process of creating and evaluating the activities and materials of a course.
Abstract: This article describes the development of an application for the grading and provision of feedback on educational processes. The too, named EduZinc, enables instructors to go through the complete p...
9 citations
14 Nov 2016
TL;DR: This paper explores how to automatically identify error classes by clustering a set of submitted codes, using code plagiarism detection tools to measure the similarity between the codes.
Abstract: Online platforms to learn programming are very popular nowadays. These platforms must automatically assess codes submitted by the learners and must provide good quality feedbacks in order to support their learning. Classical techniques to produce useful feedbacks include using unit testing frameworks to perform systematic functional tests of the submitted codes or using code quality assessment tools. This paper explores how to automatically identify error classes by clustering a set of submitted codes, using code plagiarism detection tools to measure the similarity between the codes. The proposed approach and analysis framework are presented in the paper, along with a first experiment using the Code Hunt dataset.
8 citations
Cites methods from "Pythia reloaded: an intelligent uni..."
...The correctness can, for example, be assessed using a unit testing approach such as the one used in previous work [4]....
[...]
...This paper builds upon a previous work that developed an intelligent unit testing-based code grader for education [3, 4]....
[...]
01 Jul 2017
TL;DR: Training materials built from contest tasks to teach and learn how to design algorithms that solve concrete and contextualised problems are presented.
Abstract: It is important for a future computer science engineer or scientist to master algorithm design and to know how to optimise algorithms to solve real-world problems. Most programming and IT contests require their contestants to design algorithms to solve problems and to optimise their code to get the best temporal and spatial performances. This paper presents training materials built from contest tasks to teach and learn how to design algorithms that solve concrete and contextualised problems. The first learning modules will be built thanks to a pedagogical device that will be deployed during the 2017–2018 academic year at ECAM in the frame of the LADO project. All the produced materials will be open sourced and available in English.
5 citations
13 Jul 2019
TL;DR: A platform that has been designed to browse a database of resources that can be used to teach or to learn computer science and can be searched efficiently thanks to the proposed structure for its content.
Abstract: Nowadays, teaching and learning computer science is done at various ages, for several topics and for different reasons. Depending on the country, it can start from the primary school and it finishes at the higher education level, or even later if we take continuing education into consideration. Topics to be learned can be as simple as binary representation or basic programming concepts that can be taught to children to introduce them to computer science. It is also possible to teach and learn advanced data structures or algorithms optimisation, which are interesting skills for Olympiad in Informatics contestants, for example. Recently, there is a prominent number of websites and applications that have been created to help the teaching and learning of many informatics concepts. This paper presents a platform that has been designed to browse a database of resources that can be used to teach or to learn computer science. This digital library contains freely accessible resources and can be searched efficiently thanks to the proposed structure for its content. It has been designed to maximise the user’s experience and to fit modern models of digital libraries. For each resource, a detailed information sheet has been produced, containing among other things pedagogical information to help teachers and learners use the resources as best as possible. This platform can also be used to train candidates to Olympiad in Informatics and other related and similar competitions.
3 citations
TL;DR: The study is a confirmation of previous work stating that automatic assessment may be less reliable for students with lower marks, but more trustworthy for the high achieving students and practitioners should be aware of the drawbacks of automatic assessment before choosing it.
Abstract: Aim/Purpose The aims of this study were to investigate the feasibility of automatic assessment of programming tasks and to compare manual assessment with automatic assessment in terms of the effect of the different assessment methods on the marks of the students. Background Manual assessment of programs written by students can be tedious. The assistance of automatic assessment methods might possibly assist in reducing the assessment burden, but there may be drawbacks diminishing the benefits of applying automatic assessment. The paper reports on the experience of a lecturer trying to introduce automated grading. Students’ solutions to a practical Java programming test were assessed both manually and automatically and the lecturer tied the experience to the unified theory of acceptance and use of technology (UTAUT). Methodology The participants were 226 first-year students registered for a Java programming course. Of the tests the participants submitted, 214 were assessed both manually and automatically. Various statistical methods were used to compare the manual assessment of student’s solutions with the automatic assessment of the same solutions. A detailed investigation of reasons for differences was also carried out. A further data collection method was the lecturer’s reflection on the feasibility of automatic assessment of programming tasks based on the UTAUT. Contribution This study enhances the knowledge regarding benefits and drawbacks of automatic assessment of students’ programming tasks. The research contributes to the UTAUT by applying it in a context where it has hardly been used. Furthermore, the study is a confirmation of previous work stating that automatic assessment may be less reliable for students with lower marks, but more trustworthy for the high achieving students. Automatic Assessment of Programs 202 Findings An automatic assessment tool verifying functional correctness might be feasible for assessment of programs written during practical lab sessions but could be less useful for practical tests and exams where functional, conceptual and structural correctness should be evaluated. In addition, the researchers found that automatic assessment seemed to be more suitable for assessing high achieving students. Recommendations for Practitioners This paper makes it clear that lecturers should know what assessment goals they want to achieve. The appropriate method of assessment should be chosen wisely. In addition, practitioners should be aware of the drawbacks of automatic assessment before choosing it. Recommendation for Researchers This work serves as an example of how researchers can apply the UTAUT theory when conducting qualitative research in different contexts. Impact on Society The study would be of interest to lecturers considering automated assessment. The two assessments used in the study are typical of the way grading takes place in practice and may help lecturers understand what could happen if they switch from manual to automatic assessment. Future Research Investigate the feasibility of automatic assessment of students’ programming tasks in a practical lab environment while accounting for structural, functional and conceptual assessment goals.
3 citations
Cites methods from "Pythia reloaded: an intelligent uni..."
...For lecturers, the development of test suites is the instrument to provide relevant feedback to the students, which is directly related to the code they are writing and to the test case that failed (Combéfis & Paques, 2015)....
[...]
References
More filters
Book•
01 Jan 1999
TL;DR: You may love XP, or you may hate it, but Extreme Programming Explained will force you to take a fresh look at how you develop software.
Abstract: Software development projects can be fun, productive, and even daring. Yet they can consistently deliver value to a business and remain under control.Extreme Programming (XP) was conceived and developed to address the specific needs of software development conducted by small teams in the face of vague and changing requirements. This new lightweight methodology challenges many conventional tenets, including the long-held assumption that the cost of changing a piece of software necessarily rises dramatically over the course of time. XP recognizes that projects have to work to achieve this reduction in cost and exploit the savings once they have been earned.Fundamentals of XP include: Distinguishing between the decisions to be made by business interests and those to be made by project stakeholders. Writing unit tests before programming and keeping all of the tests running at all times. Integrating and testing the whole system--several times a day. Producing all software in pairs, two programmers at one screen. Starting projects with a simple design that constantly evolves to add needed flexibility and remove unneeded complexity. Putting a minimal system into production quickly and growing it in whatever directions prove most valuable.Why is XP so controversial? Some sacred cows don't make the cut in XP: Don't force team members to specialize and become analysts, architects, programmers, testers, and integrators--every XP programmer participates in all of these critical activities every day. Don't conduct complete up-front analysis and design--an XP project starts with a quick analysis of the entire system, and XP programmers continue to make analysis and design decisions throughout development. Develop infrastructure and frameworks as you develop your application, not up-front--delivering business value is the heartbeat that drives XP projects. Don't write and maintain implementation documentation--communication in XP projects occurs face-to-face, or through efficient tests and carefully written code.You may love XP, or you may hate it, but Extreme Programming Explained will force you to take a fresh look at how you develop software. 0201616416B04062001
6,030 citations
"Pythia reloaded: an intelligent uni..." refers methods in this paper
...CONCLUSION Unit testing framework is generally used by developers in test-driven development [1], for example....
[...]
Book•
01 Nov 2004
TL;DR: Whether you have a small team that is already closely aligned with your customers or a large team in a gigantic or multinational organization, you will find a wealth of ideas to challenge, inspire, and encourage you and your team members to substantially improve your software development.
Abstract: Whether you have a small team that is already closely aligned with your customers or a large team in a gigantic or multinational organization, you will find in these pages a wealth of ideas to challenge, inspire, and encourage you and your team members to substantially improve your software development.You will discover how to: Involve the whole team-XP style Increase technical collaboration through pair programming and continuous integration Reduce defects through developer testing Align business and technical decisions through weekly and quarterly planning Improve teamwork by setting up an informative, shared workspaceYou will also find many other concrete ideas for improvement, all based on a philosophy that emphasizes simultaneously increasing the humanity and effectiveness of software development.Every team can improve. Every team can begin improving today. Improvement is possible-beyond what we can currently imagine. Extreme Programming Explained, Second Edition, offers ideas to fuel your improvement for years to come.
1,142 citations
16 Jun 2013
TL;DR: A simple language for describing error models in terms of correction rules is introduced, and a rule-directed translation strategy is formally defined that reduces the problem of finding minimal corrections in an incorrect program to the problems of synthesizing a correct program from a sketch.
Abstract: We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student's incorrect solutions, providing them with a measure of exactly how incorrect a given solution was, as well as feedback about what they did wrong.We introduce a simple language for describing error models in terms of correction rules, and formally define a rule-directed translation strategy that reduces the problem of finding minimal corrections in an incorrect program to the problem of synthesizing a correct program from a sketch. We have evaluated our system on thousands of real student attempts obtained from the Introduction to Programming course at MIT (6.00) and MITx (6.00x). Our results show that relatively simple error models can correct on average 64% of all incorrect submissions in our benchmark set.
398 citations
"Pythia reloaded: an intelligent uni..." refers methods in this paper
...proposes a method to automatically provide feedback for introductory programming problems [8]....
[...]
10 Oct 2000
TL;DR: The capabilities of a user-mode virtual machine, the design and implementation of the port, some of the applications that it lends itself to, and some ofThe work that remains to be done are described.
Abstract: The Linux kernel has been ported so that it runs on itself, in a set of Linux processes. The result is a user space virtual machine using simulated hardware constructed from services provided by the host kernel. A Linux virtual machine is capable of running nearly all of the applications and services available on the host architecture. This paper describes the capabilities of a user-mode virtual machine, the design and implementation of the port, some of the applications that it lends itself to, and some of the work that remains to be done.
287 citations
"Pythia reloaded: an intelligent uni..." refers methods in this paper
...Pythia uses User-mode Linux [5] with a trimmed down version of ArchLinux able to boot in under one second....
[...]
26 Jun 2006
TL;DR: The initial experiences using Marmoset in several introductory computer science courses are described, from the perspectives of both instructors and students, to study and better understand the development process of students.
Abstract: We developed Marmoset, an automated submission and testing system, to explore techniques to provide improved feedback to both students and instructors as students work on programming assignments, and to collect data to perform detailed research on the development processes of students. To address the issue of feedback, Marmoset provides students with limited access to the results of the instructor's private test cases using a novel token-based incentive system. This both encourages students to start their work early and to think critically about their work. Because students submit early, instructors can monitor all students' progress on test cases, helping identify challenging or ambiguous test cases early in order to update the project specification or devote additional time in lecture or lab sessions to the difficult test cases.To study and better understand the development process of students, Marmoset can be configured to transparently capture snapshots to a central repository everytime students save their files. These detailed development histories offer a unique, detailed perspective of each student's progress on a programming assignment, from the first line of code written and saved all the way through the final edit before the final submission. This type of data has proven extremely valuable many uses, such as mining new bug patterns and evaluating existing bug-finding tools.In this paper, we describe our initial experiences using Marmoset in several introductory computer science courses, from the perspectives of both instructors and students. We also describe some initial research results from analyzing the student snapshot database.
177 citations
"Pythia reloaded: an intelligent uni..." refers background in this paper
...Finally, Marmoset is a platform that provides learners with a limited access to the teacher’s private tests to encourage them to work earlier on their assignments [9]....
[...]