International Journal of Advanced Computer Science, Vol. 1, No. 1, Pp. 30-35, Jul. 2011.

Detecting Learning Patterns during Exercise from Function Call Logs

Kohei Tanigawa, Fumiko Harada, & Hiromitsu Shimakawa
Manuscript
Received: 10,July, 2011 Revised: 13,July, 2011 Accepted: 19,July, 2011 Published: 10,August,2011

Keywords
programming education, function call logs, learning pattern, process of trial and error, education support

Abstract In programming exercises in information education, teachers have to teach, figuring out student difficulties. Today, the teachers figure out the difficulties from a source code and its execution result. However, it is difficult for the teachers to figure out the difficulties from them because they are only the results of exercises. In this paper, we propose a method to detect learning patterns during exercises by collecting and analyzing function call logs. To validate the utility of the method, we have conducted a programming exercise for 17 university students and collected function call logs. Detecting the learning patterns from them, we have realized the leaning patterns detected by the method can confine student difficulties more specifically than the learning patterns detected without the method.

coach efficiently at the time they are asked questions by the students during programming exercise. This is because the teachers cannot figure out the processes of trial and errors by students. Figure 1 shows the flow of the programming exercises.

Fig. 1. Flow of Programming Exercise

1. Introduction
To acquire programming skills, a novice programmer needs to learn describing source codes during exercise. Many colleges of information technology establish programming exercise classes in their primary stages. In the classes, students are stuck on a process of describing source codes. Students gradually acquire programming skills thorough the trial and error processes. Students ask a teacher questions on their difficulties. When the teacher supervises them, the teacher must figure out the difficulties from their source codes and execution results. However, in this way, the teacher takes too long time and it requires a lot of efforts to supervise each student. For efficient teaching, the teacher needs to figure out difficulties in which students would be stuck in advance. In this paper, we propose a method to detect frequent learning patterns from function call logs collected during an exercise. This method contributes the teacher to figure out the student difficulties from the learning patterns. The teacher can prepare efficient teaching methods corresponding each learning pattern.

In the programming exercises, as Figure 1 shows, to solve assignments which given by the teachers, the students repeat describing the source codes and executing the programs in trial and error phase. After this trial and error phase, the students who is stuck on the assignments and cannot solve by himself ask a question to the teacher. In this case, the teacher needs to figure out what was difficult for the students to understand. Therefore, using the students source codes and execution results, the teacher who is asked a questions needs to identify the causes of students difficulties. The causes of students difficulties are likely to appear in trial and error phase. If the teachers figure out the student process of trial and error, he could identify the cause of students difficulties efficiently. However, it is difficult for the teachers to guess the causes of students difficulties only from students source codes and execution results because they are produced as a result of trial and error. The teachers would waste too much time. To respond to the students questions efficiently, we need to focus on the student process of trial and error.

2. Necessity of Analyzing Learning Process

2.1. Missing processes of creating programs
In a common programming exercise, few teachers have to coach many students simultaneously. Therefore, the teachers are required to give an efficient coaching to all students in a equal way. But at present, the teachers cannot

2.2. Necessity of Classifying Process of Trial and Error

In programing exercises in information education, the similar programming assignments tend to be served every year, because learnings the students should acquire are stipulated. On similar programming assignments, the causes of students difficulties in process of trial and error are often common to many students. There are similar characteristics in the causes of many students difficulties.

Tanigawa et al.: Detecting Learning Patterns during Exercise from Function Call Logs

31

If we can classify the causes of the students difficulties and figure out as patterns, we can prepare how to coach in advance. We can improve programming coaching efficiency. However, in the present programming education, we cannot figure out this patterns accurately, because we cannot record the process of making source codes.

students need to use fundamental 3 control statements and the recording library functions to solve the assignment. In this exercise, the students aim to make a program outputting the same execution result as the teacher specifies. The execution results are displayed on the learning process recorder in response to calls of the recording library functions. At the same time, the function call logs of the recording library are recorded.

2.3. Characteristics of Function Call in Procedural Language

The programs using a procedural language like C used in novice programming exercise is described with the three fundamental control statements (sequential, selection, iteration) and function calls. The students create the programs with combining these three control statements and function calls. The way of calling functions depends on the source code the students describe. Therefore, the logs of function calls represent the characteristics of the programs the students describe. We might be able to analyze the contents and the transitions of students programs in the trial and error phase if we can record the way of calling functions as the exercise logs each time the students execute his programs.

3. Collecting Function Calls and Detecting Learning Patterns

3.1. Method Overview
In this paper, we propose a method of extracting common learning patterns by collecting and analyzing student learning processes automatically. In this method, function call logs are recorded as description processes of source codes. The function call logs are visualized using tree structured charts. The charts show diversity of learning processes. When a teacher selects an appropriate branch, the function call logs indicating student difficulties are detected as learning patterns. Figure 2 illustrates the outline of this method. This method uses a recording library to notify function calls. And a learning process recorder to output execution results and record function calls. First, the teacher prepares a recording library, a learning process recorder, and the assignment which should be solved with the recording library. Second, each student solves the assignment using them. Finally, the learning patterns common in many students are detected from analysis of function call logs collected in this exercise. By this method, the teacher can figure out the patterns of function calls many students make when they feel the assignment difficult. In this way, when the teacher will use the same assignment again, he can prepare a teaching method and teach efficiently.

Fig. 2. Outline of Method

3.3. Detecting Learning Patterns

The function call logs collected in the exercise are analyzed as that candidates of common patterns in many students are visualized as tree structured charts called learning pattern trees. The teacher selects an appropriate learning pattern tree from them. The teacher focuses on a node in the learning pattern tree. He detects the learning patterns that indicate causes of the student difficulties. The nodes which has many branches in the learning pattern tree indicates a parting point of student function calls. This nodes would reveal that students have different ideas at this point, which indicates only few students can follow the correct process. Therefore, in this method, we focus on the nodes that have many branches and its child nodes are scattered. Figure 3 illustrates the way of creating a learning pattern tree. A function call log consists of a recording function type and arguments passed to the function. We define this pair as an element. The student processes of the describing source codes are appeared in consecutive elements of the function call logs. We refer to a student consecutive elements as a calling pattern. The number written in the node of a function call tree indicates the number of students who call this consecutive element patterns. Learning pattern trees are created as follows: 1) Find the most frequent calling pattern containing 2

3.2. Collecting Function Call Logs

The students solve assignments using the recording library and the learning process recorder. When functions in the reading library are called in the student programs, the learning process recorder outputs an execution result. The
International Journal Publishers Group (IJPG)

consecutive elements from function call logs of all students. 2) Find a calling pattern containing n + 1 consecutive elements that starts previous calling pattern

32

International Journal of Advanced Computer Science, Vol. 1, No. 1, Pp. 30-35, Jul. 2011.

containing n consecutive elements. 3) Find calling patterns containing many consecutive elements by repeating the step 2 recursively. This step is repeated until the number of the students who call this calling pattern is less than a threshold . A calling pattern called by the times less than is removed from the learning pattern tree.

color, and to change line width. Every time the recording library functions are called, drawing instructions are sent to the learning process recorder. The learning process recorder draws a figure in response to the drawing instructions. Figure 5 illustrates an execution example of this programming exercise. In the source code of Figure 5, the 1 , 2 , 3 are the recording library function indicated by

Fig. 5. Programming Exercise Using Recording Functions Fig. 3. Create Function Call Tree
1 is functions. The recording library function indicated by to change the line width. It sets the line width to 10 px. The 2 is to change the recording library function indicated by line color. It sets the line color to red. The recording 3 is to draw a line. It draws a 100 function indicated by px line with 60 degree inclination. The 6 times calls of 3 makes the learning process recorder draw a function hexagon.

4. Exercise for Collecting Function Call Logs

4.1. Assignment
We conducted a programming exercise and collected function call logs to validate the utility of the proposed

5. Evaluation
5.1. Creating Learning Pattern Tree
We have created the learning pattern trees from function call logs collected from the exercise mentioned in Section 4. We define to remove learning patterns common only to a few students from the learning pattern tree. The node whose pattern is called by less than students is removed from the learning pattern tree. We created the learning pattern trees by analyzing the function call logs collected from the programming exercise mentioned in Section 4. On the tool illustrated in Figure 6, the learning pattern trees visualized in response to the threshold are displayed. The teachers set the threshold by selecting an appropriate one from the learning pattern trees that visualized by this tool. Figure 6 shows the learning pattern trees in case the threshold is 1,2,3. In the case that =1, all function call patterns are appeared in the learning pattern tree, because the function call pattern called by only one student is reflected in the learning pattern tree. For the reason, in this case, an enormous number of nodes are created by this method. On the other hand, in the case that =3, the number of the nodes created by this method is too few. The teacher may fail to detect the learning pattern cause of student difficulties. In this evaluation, =2 because the number of the nodes would be appropriate.
International Journal Publishers Group (IJPG)

Fig. 4. Assignment Figure

method. To collect function call logs, we conducted a programming exercise using recording library functions for drawing figures and a learning process recorder for displaying the execution results. On the learning process recorder, a figure is drawn in response to calling of the recording library functions prepared by the teacher. The students describe a source code using the basic 3 control statements and the recording library functions. We conducted the programming exercise for 17 university students. In the assignment, we give the figure illustrated in Figure 4 to make them describe a source code which draws the given figure.

4.2. Environment
The recording library functions used in this programming exercise are C language functions. There are recording library functions to draw a line, to change line

Tanigawa et al.: Detecting Learning Patterns during Exercise from Function Call Logs

33

Fig. 6. Learning Pattern Trees Visualized in Response to Threshold B

5.2. Learning Patterns Indicating Mistakes

Figure 7 illustrates a part of the learning pattern tree when =2. The common function call patterns are contained in the nodes of the learning pattern tree in Figure 7. The number written on a node indicates the number of students who call the consecutive function call pattern corresponding to the path from the root node to the node. There is no branching from the root to the node (a). This indicates that the 7 consecutive elements contained in node (a) is common in many students. From the branching, node (a) is found as the nearest node from the root, which may be a cause of student difficulties focusing. Nodes (b), (c), (d) and (e) are found as the one derived from node (a). From the calling pattern contained in node (b), we can detect that this is a part of a correct pattern. On the other hand, from the calling patterns contained in node (c), (d) and (e), we cannot detect whether this is correct or wrong pattern. Node (b) is found as the next branching node. Node (f), (g), (h), (i) and (j) are found as the nodes derived from node (b). From the calling pattern contained in node (f), (g) and (h), we can detect that they are parts of correct patterns. On the other hand, from the calling patterns contained in node (i) and (j), we can detect that they are wrong patterns. Comparing the branching nodes, (a) to (b), the calling pattern contained in node (b) is more useful to detect the student difficulties. Nodes (b), (c), (d) and (e) are the child nodes of node (a). The numbers of students who call these patterns are 12, 3, 2 and 2 students. Most of the students who call the pattern contained in node (a) tend to call the pattern contained in node (b). On the other hand, nodes (f), (g), (h), (i) and (j) are the child nodes of node (b). The number of students who call these patterns are 4, 4, 3, 2 and 2 students. The next calling patterns called by the students who call the pattern contained in node (b) varies widely than node (a). Therefore, the branching node with largest dispersion on the students in its child nodes appears the student difficulties accurately. We have labeled the calling pattern contained in (f), (g), (h), (i) and (j) as the learning pattern F, G, H, I and J detected by the proposed method.
International Journal Publishers Group (IJPG)

Fig. 7. Learning Pattern Tree (B=2)

5.3. Classification of Learning Pattern

We classified the learning patterns by the appearance of the execution results because the execution results are outputted as figures in this exercise. The learning patterns are classified into 3 patterns by the appearance. Figure 8 illustrates the execution results of these 3 patterns. As Figure 8 shows, we labeled these patterns as the learning patterns X, Y and Z classified by the appearance. Table 1 shows the mapping the learning patterns F, G, H, I and J into the learning patterns X, Y and Z. The learning patterns F, G, H, I and J detected by this method are classified to the learning patterns X or Y. The learning patterns F, G and H are detected as different learning patterns by this method. However, all of these patterns are classified in the learning pattern X by the appearance. Similarly, the learning patterns I and J are detected as different learning patterns by this method. However, both of these patterns are classified in the learning pattern Y by the appearance. On the other hand, the learning pattern Z is not extracted by this method.

Fig. 8. Assignment Figure

5.4. Discussion
The learning patterns F, G and H are classified into the learning pattern X by the appearance. This means the learning patterns F, G and H are the same as the execution

34

International Journal of Advanced Computer Science, Vol. 1, No. 1, Pp. 30-35, Jul. 2011.

appearance, however, the elements contained in these patterns are different. That is, this method can detect the learning pattern that is not detected by the execution appearance.
Table 1: Learning Patterns and Classification

programming exercise may be useful for the coaching to correct the students mistakes. However, these methods may not be useful for the efficient coaching with figuring out the causes of the students difficulties.

Learning pattern detected by this method F G H I J

Learning pattern classified by appearance X X X Y Y

The learning pattern Z is not detected by this method. Though the learning patterns classified into the pattern Z have the same pattern by the appearance, the elements contained in these patterns have many variations. In addition, in this case, there are no calling patterns common to more than 2 students because the pattern Z does not appear in the learning pattern tree. In the case that =1, the learning pattern Z appears as many nodes in the learning pattern tree. The learning patterns contained in these nodes are common in 1 student. This means that though the calling patterns classified into the learning pattern Z have the same execution appearance, they have a wide variety of causes. They cannot be extracted as frequent patterns. When the teacher teaches the students mistaking in these patterns, he cannot figure out the student difficulties from the execution results. The teaching relying on the execution results could be cause of student confusion. Figure 9 illustrates the relationship of the two classification approaches. Different granularity patterns are detected from the two classification approaches. The learning patterns detected by this method can confine the student difficulties more specifically than the learning patterns classified by the appearance. With this method, using the learning patterns common in the many students, the teacher can figure out the student difficulties precisely. Therefore, the teacher can prepare the teaching method before the exercise corresponding to the learning patterns.

Fig. 9. Relationship of Two Classification Approaches

6. Related Works
There are some methods to figure out the students conditions of comprehension from the students mistakes during programming exercise[1],[2],[3]. In the method[1], analyzing the Java error messages, we can detect the error patterns the students often be stuck. But, we cannot detect the algorithmic error because the method presented in [1] analyzes the compile error messages. In [2], matching the syntax patterns of the students source codes during programming exercise, we can detect the bugs that compiler cannot detect. But, the method cannot detect bugs the teachers cannot predict because it uses static matching. In [3], analyzing the syntax mistakes of students source codes and the compile error message by using Bayesian network, we can predict the students thinking way stochastically. The methods analyzing the students mistakes during the

The method proposed in [4] records the number of the students source codes lines, execution times, the transition of the number of variables and syntax during the trial and error. We can manage the students progresses during the exercise. The method can detect that the students are stuck on the exercise from decreasing of the students progresses. But, this method cannot detect the causes of the students difficulties. The studies to analyze the process of describing the source codes have been in the field of Program Comprehension[ 5],[6],[7]. In these studies, they record the changes of the source codes during describing and analyze the process of creating the programs. Especially [5] records the all operations effecting the source codes and replicate the process of creating programs in detail. If we apply the method to the programming exercise, we could detect the causes of certain student difficulties pursuing his process of trial and error. However, using this method, we cannot figure out the causes of the students difficulties as patterns and make efficient coaching. Our method collect the function call logs from process of describing the source codes and can detect the causes of difficulties common in many students during process of trial and error as patterns. Using our method, the teacher can figure out the students learning processes and coach the students efficiently.

7. Conclusion
In this paper, we proposed a method of detecting the common learning patterns during exercises by collecting and analyzing function call logs. Analysis focusing on function call logs enables us to detect the common learning patterns that indicate causes of the student difficulties. Using this method, the teacher can prepare teaching methods in response to the learning patterns in advance. When the teacher will use same assignment again in another class, he can teach efficiently. We conducted a programming exercise for 17 university
International Journal Publishers Group (IJPG)

Tanigawa et al.: Detecting Learning Patterns during Exercise from Function Call Logs

35

students and collected function call logs. We can extract the learning patterns that indicate causes of the student difficulties from them. We realized that the leaning patterns detected by the method can confine the student difficulties specifically. As a future work, we will devise another exercise this method can be applied to.

MSR 05: Proceedings of the 2005 international workshop on Mining software repositories, pp.1-5, New York, USA, ACM.

References
[1] M. Hristova, A. Misra, M. Rutter, & R. Mercuri, Identifying and correcting java programming errors for introductory computer science students, (2003) SIGCSE 03: Proceedings of the 34th SIGCSE technical symposium on Computer science education, pp.153-156, New York, USA, ACM. M. Oda, & T. Kakeshita, pitfall detect in of c programs using pattern matching, (1994) Transactions of Information Processing Society of Japan, vol.35, no.11, pp.2427-2436,. J. Kasurinen, & U. Nikula, Estimating programming knowledge with Bayesian knowledge tracing, (2009) ITiCSE 09: Proceedings of the 14th annual ACM SIGCSE conference on Innovation and technology in computer science education, pp.313-317, New York, USA, ACM. R. Nakamura, T. Nishida, & T. Matsuura, The Programming Environment for Novice Education, (2005) IPSJ SIG Technical Reports, vol.2005, no104, pp65-71. T. Omori, & K. Maruyama, A change-aware development environment by recording editing operations of source code, (2008) MSR 08: Proceedings of the 2008 international working conference on Mining software repositories, pp.31-34, New York, USA, ACM. M. Kim, & D. Notkin, Program element matching for multi-version program analyses, (2006) MSR 06: Proceedings of the 2006 international workshop on Mining software repositories, pp.58-64, New York, USA, ACM. I. Neamtiu, J.S. Foster, & M. Hicks, Understanding source code evolution using abstract syntax tree matching, (2005)

Kohei Tanigawa was born in Japan in 1987. He is a student of the Graduate School of Science and Engineering, Ritsumeikan University.

[2] [3]

Fumiko Harada was born in 1980. She obtained her Ph.D. degree in Engineering from Osaka University, Japan in 2007. She has been a university lecturer of Ritsumeikan University, in Japan.

[4] [5]

[6]

[7]

Hiromitsu Shimakawa was born in 1961.He obtained his Ph.D. degree in Engineering from Kyoto University, Japan in 1999. From 2002, he has been a professor of Ritsumeikan University in Japan. His research interests include data mining and user interface. He is also interested in programming education. He is a member of IEEE and ACM.

International Journal Publishers Group (IJPG)