As computing becomes a mainstream discipline embedded in the school curriculum and acts as an enabler for an increasing range of academic disciplines in higher education, the literature on introductory programming is growing. Although there have been several reviews that focus on specific aspects of introductory programming, there has been no broad overview of the literature exploring recent trends across the breadth of introductory programming. This paper is the report of an ITiCSE working group that conducted a systematic review in order to gain an overview of the introductory programming literature. Partitioning the literature into papers addressing the student, teaching, the curriculum, and assessment, we explore trends, highlight advances in knowledge over the past 15 years, and indicate possible directions for future research.

/pdf/introductory-programming-a-systematic-literature-review-3yllvt7k18.pdf

Introductory programming: a systematic literature review

We present GradeIT, a system that combines the dual objectives of automated grading and program repairing for introductory programming courses (CS1). Syntax errors pose a significant challenge for testcase-based grading as it is difficult to differentiate between a submission that is almost correct and has some minor syntax errors and another submission that is completely off-the-mark. GradeIT also uses program repair to help in grading submissions that do not compile. This enables running testcases on submissions containing minor syntax errors, thereby awarding partial marks for these submissions (which, without repair, do not compile successfully and, hence, do not pass any testcase). Our experiments on 15613 submissions show that GradeIT results are comparable to manual grading by teaching assistants (TAs), and do not suffer from unintentional variability that happens when multiple TAs grade the same assignment. The repairs performed by GradeIT enabled successful compilation of 56% of the submissions having compilation errors, and resulted in an improvement in marks for 11% of these submissions.

Automatic Grading and Feedback using Program Repair for Introductory Programming Courses

Practical programming competencies are critical to the success in computer science (CS) education and go-to-market of fresh graduates. Acquiring the required level of skills is a long journey of discovery, trial and error, and optimization seeking through a broad range of programming activities that learners must perform themselves. It is not reasonable to consider that teachers could evaluate all attempts that the average learner should develop multiplied by the number of students enrolled in a course, much less in a timely, deep, and fair fashion. Unsurprisingly, exploring the formal structure of programs to automate the assessment of certain features has long been a hot topic among CS education practitioners. Assessing a program is considerably more complex than asserting its functional correctness, as the proliferation of tools and techniques in the literature over the past decades indicates. Program efficiency, behavior, and readability, among many other features, assessed either statically or dynamically, are now also relevant for automatic evaluation. The outcome of an evaluation evolved from the primordial Boolean values to information about errors and tips on how to advance, possibly taking into account similar solutions. This work surveys the state of the art in the automated assessment of CS assignments, focusing on the supported types of exercises, security measures adopted, testing techniques used, type of feedback produced, and the information they offer the teacher to understand and optimize learning. A new era of automated assessment, capitalizing on static analysis techniques and containerization, has been identified. Furthermore, this review presents several other findings from the conducted review, discusses the current challenges of the field, and proposes some future research directions.

Automated Assessment in Computer Science Education: A State-of-the-Art Review

Programming assignment grading can be time-consuming and error-prone if done manually. Existing tools generate feedback with failing test cases. However, this method is inefficient and the results are incomplete. In this paper, we present AutoGrader, a tool that automatically determines the correctness of programming assignments and provides counterexamples given a single reference implementation of the problem. Instead of counting the passed tests, our tool searches for semantically different execution paths between a student's submission and the reference implementation. If such a difference is found, the submission is deemed incorrect; otherwise, it is judged to be a correct solution. We use weakest preconditions and symbolic execution to capture the semantics of execution paths and detect potential path differences. AutoGrader is the first automated grading tool that relies on program semantics and generates feedback with counterexamples based on path deviations. It also reduces human efforts in writing test cases and makes the grading more complete. We implement AutoGrader and test its effectiveness and performance with real-world programming problems and student submissions collected from an online programming site. Our experiment reveals that there are no false negatives using our proposed method and we detected 11 errors of online platform judges.

https://par.nsf.gov/servlets/purl/10087529

Automatic grading of programming assignments: an approach based on formal semantics

A broad review of research on the teaching and learning of programming was conducted by Robins et al. in 2003. Since this work there have been several reviews of research concerned with the teaching and learning of programming, in particular introductory programming. However, these reviews have focused on highly specific aspects, such as student misconceptions, teaching approaches, program comprehension, potentially seminal papers, research methods applied, automated feedback for exercises, competency-enhancing games, and program visualisation. While these aspects encompass a wide range of issues, they do not cover the full scope of research into novice programming. Some notable areas that have not been reviewed are assessment, academic integrity, and novice student attitudes to programming. There does not appear to have been a comprehensive review of research into introductory programming since that of Robins et al. It is therefore timely to conduct and present such a review in order to gain an understanding of the research focuses, to highlight advances in knowledge since 2003, and to indicate possible future directions for research. The working group will conduct a systematic literature review based on the guidelines proposed by Kitchenham et al. This research project is well suited to an ITiCSE working group as the synthesis and discussion of the literature will benefit from input from a variety of researchers drawn from different backgrounds and countries.

/pdf/a-review-of-introductory-programming-research-2003-2017-3wt9alnj1b.pdf

A review of introductory programming research 2003–2017

Using automated grading tools to provide feedback to students is common in Computer Science education. The first step of automated grading is to find defects in the student program. However, finding bugs in code has never been easy. Comparing computation results using a fixed set of test cases is still the most common way to determine correctness among current automated grading tools. It takes time and effort to design a good set of test cases that can test the student code thoroughly. In practice, tests used for grading are often insufficient for accurate diagnosis.In this paper, we present our utilization of industrial automated testing on student assignments in an introductory programming course. We implemented a framework to collect student codes and apply industrial automated testing to their codes. Then we interpreted the results obtained from testing in a way that students can understand easily. We deployed our framework on five different introductory C programming assignments here at the University of Illinois at Urbana-Champaign.The results show that the automated feedback generation framework can discover more errors inside student submissions and can provide timely and useful feedback to both instructors and students. A total of 142 missed bugs were found within 446 submissions. More than 50% of students received their feedback within 3 minutes of submission.

Bei Pang

Papers

Automated Feedback Framework for Introductory Programming Courses