Towards a Framework for Mining Students

Programming Assignments

Ella Albrecht, Jens Grabowski

Institute of Computer Science
University of Gttingen
Gttingen, Germany
ella.albrecht@cs.uni-goettingen.de

Abstract Due to an increasing number of students, more
and more learning institutions tend to use computer-supported
learning tools like online learning platforms or intelligent
tutoring systems. This has opened up the opportunity to collect a
huge amount of students data. Educational Data Mining (EDM)
uses mining techniques to derive information from these data
about students knowledge, behavior and experience to improve
education. In this paper, we present a framework for mining
programming errors of computer science students by analyzing
the students solutions to a programming assignment. The
framework serves as both, a computer aided assessment tool as
well as an immediate feedback tool about the learning progress of
the students for the educator.
Keywordscomputer aided assessment; educational data
mining; programming errors
I. INTRODUCTION
Assessment is a useful mean to provide feedback to
students as well as to educators about the learning progress.
However, providing qualitative feedback is a time consuming
task. Especially the assessment of programming exercises is
exceedingly hard, since the number of all possible solutions is
unlimited and student numbers increase. Often programming
exercises are assessed by using black box tests which compare
the students solution outputs with the expected output to
determine whether the solutions are correct. The disadvantage
of this practice is that black box tests do not deliver further
information about the errors made. So either the students get no
concrete information about their faults at all, or the teaching
staff has to go through the solutions manually to provide
valuable feedback.
In the last years, many institutes register an increasing
number of beginning students. Way above 200 students have
enrolled in computer science in the winter semester 2015/16 at
the University of Gttingen. That is about 65% more than the
year before. This hampers a manual assessment. To cope with
the amount of students, a variety of automatic assessment tools
was developed [1].
Recent computer aided assessment (CAA) tools like, e.g.,
ASSYST [2], are not only able to check the correctness of a
program, but also its complexity, efficiency and style.
Furthermore, the tools try to give students enhanced feedback
to their mistakes. For example JACK [3], a web-based system
for the assessment of Java programs, offers feedback in form of
run time traces [4] and graph representations of data structures
[5].
To investigate common student programming errors,
various studies were carried out. They differ in the investigated
programming language and in the kind of information about
mistakes and how it was obtained. One of the first studies on
first-year students mistakes [6] used goal/plan analysis to find
logical errors in Pascal programs and to suggest possible
misconceptions the students may have had. Further attempts to
gather common programming errors were the evaluation of
compiler error logs [7][9], observations by the teaching staff
during class [10], and interviews and surveys with students and
educators [7], [11].
CAA has opened the possibility to collect a huge amount of
students data, this opportunity is not fully exploited. Most
CAA tools just assess a students solution and provide some
kind of feedback. Data gained during the assessment process,
e.g., compiler logs, traces, or code metrics, can be used to
detect which students require support or predict their
performance. However, the generated data are often not
processed and thus remain unused [12].
In this paper, we present a framework which serves as a
CAA tool for programming assignments. In contrast to many
existent CAA tools, it exploits the data generated during the
analysis of the students solutions to draw conclusions about
students programming behavior and skills. The paper is
organized as follows. In Section 2, we introduce our research
project and the basic structure of the project. The particular
phases and the current status of the project are then discussed
in Section 3. Section 4 deals with the implementation details of
a prototype of the framework for the analysis of students C
programs.
II. PROJECT AND FRAMEWORK STRUCTURE
In our project, we are developing a framework that aims at
collecting and processing data about students solutions to a
programming assignment. It serves as both, a computer aided
assessment tool as well as a mining platform which provides
immediate feedback about the learning progress of the students
to the educator. The collected data can be examined to make
conclusions about students programming behavior and skills.
The data are composed of the students programming errors, an

978-1-4673-8633-3/16/$31.00 2016 IEEE 10-13 April 2016, Abu Dhabi, UAE

2016 IEEE Global Engineering Education Conference (EDUCON)
Page 1096
evaluation of the programming style, and several software
metrics.
A. Programming Errors
The core element of the framework is the detection,
localization, and classification of programming errors. There
exist multiple classification schemes for errors [13]. In our
work, we divide the errors into syntax, semantic and logic
errors. Syntax errors are errors due to the violation of
grammatical rules of the programming language, e.g., a
missing semicolon or omitted or misplaced braces. All syntax
errors can be detected by a compiler. When the compiler finds
an error, it returns an error message with the reason for the
error and the line number at which the error is located.
A semantic error occurs if programming constructs are used
wrong even if the program may be syntactically correct. We
can distinguish between static and dynamic semantic errors.
Static semantic errors are semantic errors that are detected by a
compiler, e.g., uninitialized variables or type mismatches.
Dynamic semantic errors (or runtime errors) are errors which
occur during the execution of a program, e.g., index out of
bounds or null pointer exception. Some dynamic semantic
errors can be detected before execution by static analysis tools.
Syntax as well as semantic errors lead to a non-execution or an
interruption of the execution of a program.
Logic errors cause that a program is not doing what it is
intended to do. This error type is independent of the
programming language and occurs when the programmer
makes a mistake in the design of his program or has
implemented an algorithm incorrectly. Typical logic errors are,
e.g., off-by-one errors or unregarded cases. Logic errors are
most difficult to find and can only be revealed by testing.
B. Basic Structure of the Framework
Fig. 1 gives an overview of the basic structure of the
framework. At first, the students solutions are analyzed
statically by using compilers and other static analysis tools to
detect syntax and semantic errors. Those solutions that are
compiled successfully are then analyzed dynamically by using
unit or black box tests. These tests determine which of the
solutions are correct. Afterwards, the compilable solutions are
clustered by similarity where each cluster has to contain at least
one correct solution. The correct solutions are used to compare
them with the students solutions in the same cluster to localize
logic errors. Errors found during static analysis and the ones
localized during program comparison are classified into error
types and stored in a database. Results from the dynamic
analysis are also stored in the database, such that the data can
be used to provide enhanced feedback to students as well as to
educators.
III. PROJECT PHASES
For our project, we have chosen an incremental
development process which proceeds in three phases. In Fig. 1,
parts of the framework which will be added not until the
second phase of our project are shaded in gray. Dashed objects
are parts which are worked on in the first as well as in the
second phase. Since the third phase only involves evaluation, it
is not depicted in the figure.
978-1-4673-8633-3/16/$31.00 2016 IEEE
2016 IEEE Global Engineering Education Conference (EDUCON)
Page 1097
A. Phase 1 Basic Static and Dynamic Analysis
In the first phase, we use conventional program analysis
methods to study students solutions. For the identification of
syntax and semantic errors, the results of different compilers
and static analysis tools are parsed. The found errors are then
assigned to the corresponding error types. We collect errors as
well as warnings, since a warning may give a hint on a
semantic error. For example, for the program in Fig. 2 a
compiler may return the warning message:
s5.c:13:10: warning: if statement has empty body [-
Wempty-body]
if (i>0);
^
The message signifies a semicolon directly after the if-
condition in line 10, and therefore the if-statement has an
empty body. Most often, it is not the intention of programmers
to close the statement in such a case, but rather they intended to
include the subsequent instructions in the block of the if-
statement.
When students start to program, they often only focus on
the functionality of the program. Missing indentation or odd
variable names make it hard to read and understand the code in
most cases. It is important that students learn to write not only
correct but also readable programming code. Programming
style is composed of two aspects, the compliance of special
code conventions and the personal style of the programmer,
e.g., preferred programming structures, length of variable
names or lines, number of variables, and usage of blank lines.
Several style checking tools [14][16] exist to automatically
validate whether a code complies the agreed conventions. For
the investigation of the students personal programming style,
we collect various code metrics like, e.g., the ones used in [17]
and [18]. We collect code metrics for the analysis of the
complexity of the students solutions and for the calculation of
the similarity as part of our static analysis.
The steps after the static analysis are only executed for
solutions that were compiled successfully, since the subsequent
steps require execution of the students solution. To check
whether a solution is correct, we use black box or unit tests,
depending on whether students had to implement a whole
program or just a function. These tests consist of different test
cases, defining input parameters and expected output of the
program resp. the function. Furthermore, we collect
information about the code coverage for each test case and
measure the execution time.
Feedback regarding the mistakes students made in their
solutions is crucial to improve their knowledge. But also for the
educators it is helpful to know with which problems their
students are struggling. Therefore, our framework provides
feedback to students as well as to educators. In the first project
phase, the feedback that is given to the students and the
educators consists of raw results of the different tools used to
evaluate the solution resp. simple statistics about the errors
made. To draw conclusions from the made errors, we need to
classify them. Since the same errors can be detected with
different tools, a mapping between specific errors of the
particular tools to defined error types is needed, so that the
same mistakes are not counted multiple times. For example, for
10-13 April 2016, Abu Dhabi, UAE
the program from Fig. 2 a compiler as well as a static analysis
tool may state that the array in line 8 is out of bounds.
The presented methods are used in most assessment tools
and have been demonstrated to be sufficient to assess
programming assignments in a simple way. At the end of the
first phase which will be in February/March 2016, the
framework will be used the first time in an introductory C
course at the University of Gttingen. Participants of the course
are first-year students in computer science, mathematics,
physics, biology and economics. Data collected during this
course is used to enhance the framework. This extension will
be done in the second phase of the project.
B. Phase 2 Localization of Logic Errors
Since black box tests can only tell whether a programs
output is incorrect but not why, we extend our framework by a
deeper analysis which aims at the localization of logic errors in
the second phase of the project.
The detection of logic errors requires knowledge about
what the program is intended to do. The easiest way for an
educator to describe the intention of a program is to provide a
sample solution. In general, the problem whether two programs
are semantically equivalent, i.e., have the same intention and
behavior, is undecidable [19]. But as student programs are
often very simple and have not that many variations,
comparison to a correct solution can reveal logic errors.
The more correct solutions for comparison are available,
student solutions
static analysis
students x compilation
x
x
static code
analysis
style
x code metrics
exercise
educator
clustering
sample
solution
feedback
Fig. 1. Basic structure of the framework
978-1-4673-8633-3/16/$31.00 2016 IEEE
2016 IEEE Global Engineering Education Conference (EDUCON)
Page 1098
the higher is the probability to detect all logic errors. However,
comparison of programs to detect logic errors is time-
consuming, since, e.g., variables can be named different and
have to be matched, control flow graphs, abstract syntax trees
or traces have to be compared. To reduce this effort, we cluster
the solutions by similarity, because it is easier and more
probable to localize logic errors in programs which are similar.
Since we want to compare a students solution with a correct
solution, each cluster needs to contain at least one correct
solution. After the clustering, we compare each solution within
a cluster with the correct solutions in the cluster.
The investigation of convenient similarity metrics and
program comparison techniques will be part of the second
phase of our project. Moreover, we will study the most
common errors made by the students to improve the feedback.
Students shall not only get raw results, but also hints why the
error appears and how it could be solved. For educators the
feedback is enhanced, such that they can also see the evolution
of the different error types and if prior misunderstandings of
particular programming concepts could be managed
successfully. The second phase is planned to last from April
2016 till March 2017 and will be completed with a further
application in the introductory C course.
C. Phase 3 Evaluation
The third phase deals with the evaluation of the framework.
We analyze the precision and the miss rate of the framework as
well as its impact on the added value for the students and for
feedback
compilation ok? black box tests
U
D
D
U
D
error classification
results
error localization
error localization
error localization
10-13 April 2016, Abu Dhabi, UAE
 1: #include <stdio.h>
2: #include <limits.h>
3:
4: int main(void) {
5: char c = 'A';
6: int i;
7: int arr[5];
8: int a = arr[-4];
9: char *f (const char *s);
10: if (i>0);
11: while (c <= 'z') {
12: printf("%c", c);
13: c = c + 1;
14: }
15: return 0;
16: }

Fig. 2. Example for a students program that aims at printing out all
characters between A and z

the educator. Furthermore, we will use the gained data to

investigate programming beginners change in experience
concerning programming style and the amount and types of
errors made.
D. Current status of the Project
At the moment, we are in the first phase of our project. We
have implemented a prototype of the framework. Several tools
were integrated to perform a basic analysis of the students
solutions. Details on the implementation are given in Section
IV. Furthermore, we have prepared the exercises and the
required test files for black box tests for the C course. In a
preliminary study, students have solved the exercise and tested
their programs using the provided test files. The outcome was
that the exercises can be solved in an adequate time by
programming beginners. In a few exercises, some students had
problems to understand what is actually demanded, so we
adjusted the formulation of the task accordingly. Furthermore,
we corrected some minor errors in the test files.
IV. IMPLEMENTATION
We have implemented a prototype of the framework for the
analysis of students C programs. Fig. 3 shows the preference
window for a new evaluation. The educators first need to
choose a programming language, before available options are
shown. At the moment only C is support as language, but it is
planned to also include Java. The educator can choose a course
and enter an exercise name and exercise number. The educators
also have to define the folder which contains files needed for
test execution, e.g. dejagnu or cunit files which define
testcases, and the destination folder on a virtual machine (VM)
where the testfiles and students solution has to be copied to
and on which the dynamic tests will be executed. Furthermore,
they can choose which tests shall be executed for the exercise.
The students solution are imported via a zip file with the
following folder structure:
student/submissionNo/
where student is a unique identifier for a student and
submissionNo is the number of the solution submitted by this
student, if multiple submissions are allowed. We need this
information when we want to investigate the evolution of a
students programming experience.
Fig. 3. Preference window for a new evaluation
For compilation we use gcc [20] and clang [21]. We use
two different compilers because we have observed that they
return different results regarding semantic checks and
warnings. As further static analysis tools we have integrated
splint [22] which is a tool for checking C programs for security
vulnerabilities and coding mistakes. The drawback of splint is
its high false positive rate. As a second static code analysis
tool, we integrated cppcheck [23] which does not detect as
many errors as splint, but has a lower false positive rate. To
check whether code conventions are complied, we use vera++
[24]. It provides a set of rules for checking the style of the
program, but it also offers the possibility to formulate own
rules in Tcl (Tool Command Language). The output of these
tools is written in text or XML files. We parse those files and
lookup for the corresponding error types in the database. The
identification of the error types depends on the tools, e.g.,
reported errors of gcc are matched against predefined regular
expressions to determine the error type. Cppcheck, however,
returns an XML file where each error has an attribute
containing the name of the error, so we simply have to look for
the name in the database to obtain the error type. Each error is
stored in the database with the raw error message, the line it
was found and the associated error type. Furthermore a boolean
field exists for each error, such that an educator can manually
mark an identified error as false positive.
Dynamic testing is performed in a virtual machine due to
security reasons. We use cunit [25] for testing single functions
and dejagnu [26] for testing a complete program. Dejagnu is a
framework for testing programs independently of the
programming language by comparing a programs output to an
expected output. If the execution of a test case takes longer
than 10 seconds, a timeout is reported. Since the expected
programs are very simple and should have a small execution

978-1-4673-8633-3/16/$31.00 2016 IEEE 10-13 April 2016, Abu Dhabi, UAE

2016 IEEE Global Engineering Education Conference (EDUCON)
Page 1099
time, a timeout indicates an infinite loop in most cases. The
result for each test case is stored in the database along with
associated coverage information if the coverage option was
selected. For the coverage analysis we use gcov [27] and its
graphical frontend lcov [28].
After the evaluation of students solutions the tool creates a
zip file containing an error report for each student. The report
contains the results of the different tests. For the educational
staff, the tool provides statistics about common errors for the
current exercise as well as for the overall course. Furthermore,
educators can take a look at each result for any student for any
exercise and check whether the reported errors are false
positives.
V. CONCLUSION AND FUTURE WORK
In this paper, we have presented a framework which can be
used for the automatic assessment of programming
assignments, as well as for the identification of common
programming errors made by students, and the investigation of
the evolution of programming experience. We have
implemented a prototype which is currently able to find syntax
and semantic errors by using compiler messages and static
analysis tools. Furthermore, programs are checked dynamically
by black box and unit test. We also implemented a
classification for the errors which are found by the different
tools. The classification makes it easier to investigate common
errors and to provide useful hints to the students about the
particular error.
Further research will concentrate on the detection and
classification of logic errors. AnalyseC [29] is a first attempt to
localize logic errors in programming assignments. They use
standardization, control flow, and dependence information to
compare a students solution with a model solution. In their
case study 80% of the errors could be found. We plan to extend
this approach by adding further information for the comparison
like, e.g., coverage information or traces. AnalyseC does not
classify the found errors, so we need to develop patterns for the
classification of logic errors. Furthermore, we will investigate
convenient similarity metrics for the clustering of students
solutions. A complete evaluation of the framework will be
made after using the framework in two years in the C course.
Additionally, mining techniques can be used on the data, which
is gained with the framework, to investigate the evolution of
students programming behavior, skills, and experience.
REFERENCES
[1] P. Ihantola, T. Ahoniemi, V. Karavirta, and O. Seppl, Review of
recent systems for automatic assessment of programming assignments,
in Proceedings of the 10th Koli Calling International Conference on
Computing Education Research, 2010, pp. 8693.
[2] D. Jackson and M. Usher, Grading student programs using ASSYST,
in Proceedings of the twenty-eighth SIGCSE technical symposium on
Computer science education, 1997, vol. 29, no. 1, pp. 335339.
[3] M. Goedicke, M. Striewe, and M. Balz, Computer Aided Assessments
and Programming Exercises with JACK, 2008.
[4] M. Striewe and M. Goedicke, Using Run Time Traces in Automated
Programming Tutoring, in Proceedings of the 16th Annual Joint
Conference on Innovation and Technology in Computer Science
Education, 2011, pp. 303307.
978-1-4673-8633-3/16/$31.00 2016 IEEE
2016 IEEE Global Engineering Education Conference (EDUCON)
Page 1100
[5] M. Striewe and M. Goedicke, Visualizing Data Structures in an e-
Learning System., in Proceedings of the 2nd International Conference
on Computer Supported Education, 2010, vol. 1, pp. 172179.
[6] J. C. Spohrer and E. Soloway, Novice mistakes: are the folk wisdoms
correct?, Commun. ACM, vol. 29, no. 7, pp. 624632, 1986.
[7] M. Hristova, A. Misra, M. Rutter, and R. Mercuri, Identifying and
Correcting Java Programming Errors for Introductory Computer Science
Students, ACM SIGCSE Bull., vol. 35, no. 1, pp. 153156, 2003.
[8] J. Jackson, M. Cobb, and C. Carver, Identifying Top Java Errors for
Novice Programmers, Front. Educ. 2005. FIE 05. Proc. 35th Annu.
Conf., pp. T4C24T4C27, 2015.
[9] M. Ahmadzadeh, D. Elliman, and C. Higgins, An Analysis of Patterns
of Debugging Among Novice Computer Science Students, ACM
SIGCSE Bull., vol. 37, no. 3, pp. 8488, 2005.
[10] G. L. Natalie Coull, Ishbel Duncan, Jackie Archibald, Helping Novice
Programmers Interpret Compiler Error messages, in 4th Annual LTSN-
ICS Conference, 2003.
[11] A. E. Fleury, Programming in Java: Student-Constructed Rules, in
Proceedings of the Thirty-first SIGCSE Technical Symposium on
Computer Science Education, 2000, pp. 197201.
[12] S. H. Edwards and V. Ly, Mining the Data in Programming
Assignments for Educational Research, in Proceedings of the
International Conference on Education and Information Systems:
Technologies and Applications, 2007, pp. 135140.
[13] A. J. Ko and B. A. Myers, Development and Evaluation of a Model of
Programming Errors, Proc. - 2003 IEEE Symp. Hum. Centric Comput.
Lang. Environ., 2003.
[14] Checkstyle. [Online]. Available: http://checkstyle.sourceforge.net/.
[Accessed: 16-Nov-2015].
[15] R. Jocham, JCSC - Java Coding Standard Checker. [Online].
Available: http://jcsc.sourceforge.net/. [Accessed: 16-Nov-2015].
[16] Google, Google C++ Style Guide. [Online]. Available:
https://github.com/google/styleguide/tree/gh-pages/cpplint. [Accessed:
16-Nov-2015].
[17] R. E. Berry and B. a. E. Meekings, A style analysis of C programs,
Commun. ACM, vol. 28, no. 1, pp. 8088, 1985.
[18] K. Ala-Mutka, T. Uimonen, and H.-M. Jarvinen, Supporting Students
in C++ Programming Courses with Automatic Program Style
Assessment, J. Inf. Technol. Educ., vol. 3, no. 1, pp. 245262, 2004.
[19] T. a. Budd and D. Angluin, Two Notions of Correctness and Their
Relation to Testing, Acta Inform., vol. 18, pp. 3145, 1982.
[20] I. Free Software Foundation, GCC, the GNU Compiler Collection.
[Online]. Available: https://gcc.gnu.org/. [Accessed: 16-Nov-2015].
[21] clang: a C language family frontend for LLVM. [Online]. Available:
http://clang.llvm.org/. [Accessed: 16-Nov-2015].
[22] Splint - Annotation-Assisted Lightweight Static Checking. [Online].
Available: http://www.splint.org/. [Accessed: 16-Nov-2015].
[23] Cppcheck - A tool for static C/C++ code analysis. [Online]. Available:
http://cppcheck.sourceforge.net/. [Accessed: 16-Nov-2015].
[24] Verateam, vera++. [Online]. Available:
https://bitbucket.org/verateam/vera/wiki/Home. [Accessed: 17-Nov-
2015].
[25] CUnit - A Unit Testing Framework for C. [Online]. Available:
http://cunit.sourceforge.net/. [Accessed: 16-Nov-2015].
[26] DejaGnu. [Online]. Available: http://www.gnu.org/software/dejagnu/.
[Accessed: 16-Nov-2015].
[27] gcov - a Test Coverage Program. [Online]. Available:
https://gcc.gnu.org/onlinedocs/gcc/Gcov.html. [Accessed: 16-Nov-
2015].
[28] LCOV - the LTP GCOV extension. [Online]. Available:
http://ltp.sourceforge.net/coverage/lcov.php. [Accessed: 16-Nov-2015].
[29] W. Wu, G. Li, Y. Sun, J. Wang, and T. Lai, AnalyseC: A Framework
for Assessing Students Programs at Structural and Semantic Level,
IEEE Int. Conf. Control Autom. ICCA, pp. 742747, 2007.
10-13 April 2016, Abu Dhabi, UAE