We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student's incorrect solutions, providing them with a measure of exactly how incorrect a given solution was, as well as feedback about what they did wrong.We introduce a simple language for describing error models in terms of correction rules, and formally define a rule-directed translation strategy that reduces the problem of finding minimal corrections in an incorrect program to the problem of synthesizing a correct program from a sketch. We have evaluated our system on thousands of real student attempts obtained from the Introduction to Programming course at MIT (6.00) and MITx (6.00x). Our results show that relatively simple error models can correct on average 64% of all incorrect submissions in our benchmark set.

/pdf/automated-feedback-generation-for-introductory-programming-5g15kzw2th.pdf

Automated feedback generation for introductory programming assignments

The recognition of standard computational structures (cliches) in a program can help an experienced programmer understand the program. Based on the known relationships between the cliches, a hierarchical description of the program's design can be recovered. We develop and study a graph parsing approach to automating program recognition in which programs are represented as attributed dataflow graphs and a library of cliches is encoded as an attributed graph grammar. Graph parsing is used to recognize cliches in the code.
We demonstrate that this graph parsing approach is a feasible and useful way to automate program recognition. In studying this approach, we have experimented with two medium-sized, real-world simulator programs. There are three aspects of our study.
First, we evaluate our representation's ability to suppress many common forms of program variation which hinder recognition. We find that our recognition approach is robust under syntactic, organizational, and implementational variation, as well as variation due to delocalization, unfamiliar code, and some function-sharing optimizations.
Second, we investigate the expressiveness of our graph formalism for capturing programming cliches. The formalism is able to concisely encode algorithmic and data aggregation cliches.
Third, we empirically and analytically study the computational cost of our recognition approach with respect to the real-world simulator programs. We find that although our graph parsing algorithm is exponential in the worst case, its complexity is reduced in its practical application to program recognition. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)

Automated program recognition by graph parsing

Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss’ feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.

https://dl.acm.org/doi/pdf/10.1145/3231711

A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

Automated program recognition: a feasibility demonstration

http://spider.sci.brooklyn.cuny.edu/~kopec/research/sdarticle4.pdf

Understanding and debugging novice programs

http://spider.sci.brooklyn.cuny.edu/~kopec/research/sdarticle5.pdf

LAURA, a system to debug student programs

We present in this paper an original approach of Debugging. The general goal in Automatic Verification of Programs is to prove that a program is correct or incorrect. This does not generally provide enough information to catch the bugs. On the opposite, the purpose of the debugging system LAURA, that we have designed and implemented, is to find the errors.

In order to debug a student program, the LAURA system uses a procedural description of the program task, under the form of a program model. Debugging is then viewed as a comparison of two graphs, built from the student program and from the program model. The system can apply powerful semantic transformations on the graphs to increase their resemblances and to identify subgraphs that perform a same task.

The LAURA system has shown to be able to determine the correctness of programs implemented in various ways, very different from the program model. It can also express sophisticated diagnostics and set up proper corrections.

Automatic diagnostics of semantic errors

An interactive system for understanding programs has been designed. This system provides informations about control structure and data flow. It also performs powerful semantic transformations that are checked for validity. The system relies on previously implemented algorithms that apply to a graph representation of programs.

Anne Adam

Papers

LAURA, a system to debug student programs

Automatic diagnostics of semantic errors

An interactive tool for program manipulation