Intelligent tutoring systems have become increasingly common in assisting students but are often aimed at isolated subject-domain tasks without creating a scaffolding system from lower- to higher-level cognitive skills, with low-level skills often neglected. We designed and developed an intelligent tutoring system, CompPrehension, which aims to improve the comprehension level of Bloom’s taxonomy. The system features plug-in-based architecture, easily adding new subject domains and learning strategies. It uses formal models and software reasoners to solve the problems and judge the answers, and generates explanatory feedback about the broken domain rules and follow-up questions to stimulate the students’ thinking. We developed two subject domain models: an Expressions domain for teaching the expression order of evaluation, and a Control Flow Statements domain for code-tracing tasks. The chief novelty of our research is that the developed models are capable of automatic problem classification, determining the knowledge required to solve them and so the pedagogical conditions to use the problem without human participation. More than 100 undergraduate first-year Computer Science students took part in evaluating the system. The results in both subject domains show medium but statistically significant learning gains after using the system for a few days; students with worse previous knowledge gained more. In the Control Flow Statements domain, the number of completed questions correlates positively with the post-test grades and learning gains. The students’ survey showed a slightly positive perception of the system.

Improving Comprehension: Intelligent Tutoring System Explaining the Domain Rules When Students Break Them

Intelligent tutoring systems become increasingly common in assisting human learners, but they are often aimed at isolated domain tasks without creating a scaffolding system from lower- to higher-level cognitive skills. We designed and implemented an intelligent tutoring system CompPrehension aimed at the comprehension level of Bloom’s taxonomy that often gets neglected in favour of the higher levels. The system features plugin-based architecture, easing adding new domains and learning strategies; using formal models and software reasoners to solve the problems and judge the answers; and generating explanatory feedback and follow-up questions to stimulate the learners’ thinking. The architecture and workflow are shown. We demonstrate the process of interacting with the system in the Control Flow Statements domain. The advantages and limits of the developed system are discussed.

CompPrehension - Model-Based Intelligent Tutoring System on Comprehension Level

The article discusses the problem of visualization of complex multiparameter systems, defined by datasets on their structure, functional structure, and activity in the form of complex graphs and transition of traditional representation of the data acquired by graph mining to a compact image built by pictographic methods. In these situations, we propose using the Unified Graphic Visualization of Activity (UGVA) method for data concentration and structuring. The UGVA method allows coding in an anthropomorphic image of elements of graphs with data on structural and functional features of systems and overlaying these images with the data on the system’s activity using coloring and artifacts. The image can be composed in different ways: it can include the zone of integral evaluation parameters, segmented data axes of five types, and four types of symmetry. We describe the method of creating UGVA images, which consists of 13 stages: the parametric model is represented as a structural image that is converted to a basic image that is then detailed into the particular image by defining geometric parameters of the primitives and to the individualized image with the data about a particular object. We show how the individualized image can be overlaid with the operative data as color coding and artifacts and describe the principles of interpreting UGVA images. This allows solving tasks of evaluation, comparison, and monitoring of complex multiparameter systems by showing the decision-maker an anthropomorphic image instead of the graph. We describe a case study of using the UGVA method for visualization of data about an educational process: curricula and graduate students, including the data mined from the university’s learning management system at the Siberian Federal University for students majoring in “informatics and computing”. The case study demonstrates all stages of image synthesis and examples of their interpretation for situation assessment, monitoring, and comparison of students and curricula. It allowed for finding problematic moments in learning for individual students and their entire group by analyzing the development of their competence profiles and formulating recommendations for further learning. The effectiveness of the resulting images is compared to the other approaches: elastic maps and Chernoff faces. We discuss using graph mining to generate learning problems in order to lessen the workload of gathering raw data for the UGVA method and provide general recommendations for using the UGVA method based on our experience of supporting decision making.

/pdf/evaluation-comparison-and-monitoring-of-multiparameter-2jbw4d2o.pdf

Evaluation, Comparison and Monitoring of Multiparameter Systems by Unified Graphic Visualization of Activity Method on the Example of Learning Process

The use of formal knowledge representation models in intelligent tutoring systems often requires logical reasoning on these models by predefined rules. This process can be time and memory consuming, so finding effective software reasoners for different applications is an important research field. This problem is relevant for cognitive and constraint-based intelligent tutoring systems. We performed a comparative study of various software reasoners (Pellet, Apache Jena inference subsystem, Apache Jena SPARQL query processor, SWI-Prolog with semweb package, Closed World Machine, and Answer Set Programming solvers Clingo and DLV) for solving tasks specific to intelligent tutoring systems using three formal models with different properties and corresponding rule sets created for intelligent tutoring systems in introductory programming education domain. We compared features of rule-definition formalisms for different approaches and measured run and wall time, average CPU load, and peak RAM usage based on the count of inferred RDF triples. The experiments show that Apache Jena infers the solution quicker than other reasoners on the majority of tasks but consumes a significant amount of memory, while Clingo performs significantly better for combinatorial problems.

Inference Engines Performance in Reasoning Tasks for Intelligent Tutoring Systems.

When developing automatic quizzing systems and intelligent tutoring systems, significant effort has to be spent on developing the problem bank. Question and problem generation is a field of study concerning automating this routine work. The two most common methods of problem generation are constrained random generation and template-based generation, but each of them has disadvantages. In this work, we study a possibility of generating code-tracing learning problems from existing open-source code. We generated a set of learning problems and evaluated their distinctness from human-authored problems and readiness for usage in the learning process. Both teachers and students showed the rate of determining machine-generated problems only slightly above random guessing. Teachers strongly agreed with the problems' relevance and agreed with their suitability for the learning process. Automatic labeling to filter desired problems for the assignment includes used concepts, possible errors during solving, and difficulty estimates. The studied type of learning problem required little additional data to add to the code; our further work will concern problem types with more dynamic data to overcome this limitation.

https://dl.acm.org/doi/pdf/10.1145/3545945.3569774

Generation of Code Tracing Problems from Open-Source Code

Developing an understanding of control structures is one of the important tasks in introductory programming courses. To facilitate active learning with immediate feedback, we developed a constraint-based tutor, "How it Works: Algorithms," that asks students to build an execution trace of the given algorithm and provides explanatory feedback about the mistakes the student made. The reasons for the student's faults are determined by an inference engine, using a set of rules describing the subject domain. Teachers can create exercises using a simple visual block-based interface and sending them to students as permanent links.

How it Works: Algorithms - A Tool for Developing an Understanding of Control Structures

Domain ontologies that can solve various tasks using its concepts and determine fault reason for students’ answers may serve as a good basis for creating for a testing system with extensive explanatory abilities. But capabilities of modern ontological reasoners may not be enough for this kind of task. In this study, we developed and tested an ontology able to build execution trace for the given algorithm containing sequences and alternatives and find fault reasons for incorrect traces. The study also showed problems with the used reasoner that hinder the developed ontology from becoming fully effective.

/pdf/verifying-algorithm-traces-and-fault-reason-determining-j9i3m6uf70.pdf

Verifying algorithm traces and fault reason determining using ontology reasoning.

The chapter presents the review of modern approaches for the domain terminology extraction, concept discovery, concept hierarchy derivation, and learning of non-taxonomic relations steps in the ontology learning task. The chapter presents the review of not only various approaches to solving these NLP tasks but also ready-made tools that implement these approaches.

Mikhail Denisov

Papers

Improving Comprehension: Intelligent Tutoring System Explaining the Domain Rules When Students Break Them

How it Works: Algorithms - A Tool for Developing an Understanding of Control Structures

CompPrehension - Model-Based Intelligent Tutoring System on Comprehension Level

Verifying algorithm traces and fault reason determining using ontology reasoning.

Intelligent Approaches for the Automated Domain Ontology Extraction