Eureka SuperKidz

External Evaluation: 2011-2012

Super

eureka

Kidz

Introduction
Eureka SuperKidz Program started in June 2010 NCIS, an external agency, conducted evaluation over 2 years
Baseline July 2010 End of Year 1 - April 2011 End of Year 2 April 2012

Purpose of evaluation
Evidence of value-add and impact of Eureka program

Understand areas of strength and areas that need focus

Super

eureka

Kidz

Introduction
30 Eureka and 20 Control villages selected randomly. Control villages similar in socio-economic background as Eureka villages, but without Eureka intervention.

Children from same villages and classes assessed over two years.
5991 children from pre-school to Std 8 assessed. Assessments on Tamil, Math and English skills. Pre-school children assessed on Number concepts, Cognitive and Language skills.

Super

eureka

Kidz

Tamil Evaluation Findings

There are 3 key skill levels in Tamil reading Ability to read words Ability to read sentences Ability to read stories

Super

eureka

Kidz

Tamil Evaluation Findings Eureka Initial vs Final

Children who can read Tamil words
Eureka Baseline
100 80 64 % children 60 40 34 20 3 0 27

Eureka Year 1

Eureka Year 2
90

76

Std 1

Std 2

To find out the impact of the Eureka program, we could compare how children are performing at the beginning of the program and end of each academic year.
64% Std 1 children in Eureka villages learnt to read words in year 2, compared to 27% who learnt to read words in year 1!

A comparison of childrens learning in Eureka villages alone, is not enough to conclude whether the improvement was because of the program or whether children acquired the skill over the course of a regular academic year.
5

Super

eureka

Kidz

Tamil Evaluation Findings - Need for Control Group

Control villages help us find out how much children learn in a regular academic year. We need to compare Eureka villages with Control villages to see what additional improvement there is in Eureka villages. Eureka Villages Control Villages

So what are control villages?

Control villages are similar in socio-economic background as Eureka villages Children go to similar schools But these villages do not have the Eureka program Eureka evening classes No Eureka evening classes School School

Super

eureka

Kidz

Tamil Evaluation Findings Year 1 Comparison

Children who can read words
Eureka Baseline 100 80 60 40 20 3 0 Std 1 Std 2 27 0 29 34 38 76 71 Eureka Year 1 Control Baseline Control Year 1

Eureka value add in a year = Improvement in Eureka Improvement in Control

Let us compare baseline and year 1 data for Eureka and Control. Over a period of one year, the percentage of children who could read words in Std 2 in Eureka villages, improved by 42% (=76%-34%) compared to 33% (=71%-38%) in Control villages.

The levels and progress in both Eureka and Control in Std 1 were similar.
This was an important feedback from year 1 evaluation which highlighted the need to focus on key skills for lower classes (Std 1&2). Super
eureka

Kidz

Tamil Evaluation Findings Year 1 & 2 comparison

We saw that comparing with control is important. Now, let us compare the levels across classes, both in Eureka and Control, to understand the Eureka value-add over 2 years.
100 80 64 % children 60 58 40 20 0 Std 1 Std 2 Std 3 Std 4 Std 5 Std 6 27 23

Children who can read Tamil words

Eureka Year 1 90 92 Eureka Year 2 99 97 94 95 Control Year 2 99 99 100 99 99 97

76

81

The curved lines in the graph show how children have progressed in a particular skill, across classes. We call the Control line (brown line) the Natural Progress line. The difference between Natural Progress line and the Eureka Year 2 line (blue line), shows Eurekas value add to learning improvements over 2 years.

Super

eureka

Kidz

Tamil Evaluation Findings Year 1 & 2 comparison

Children who can read Tamil words
Eureka Year 1 100 80 64 % children 60 58 40 20 0 Std 1 Std 2 Std 3 Std 4 Std 5 Std 6 27 23 90 92 Eureka Year 2 99 97 94 95 Control Year 2 99 99 100 99 99 97

76

Eureka Value-Add

81

Atleast 30-40% more children in Std 1&2 in Eureka, can read words compared to control. Without Eureka, only 23% (as per natural progress) would possibly have read words!

Last years evaluation had showed the need to focus on important skills like word reading for lower classes. Focusing on this skill in Eureka villages in year 2, helped many who could read letters, to learn to read words.
By getting children to read words, as expected in lower classes, we can help them progress in their reading ability (to read sentence or story) in subsequent classes.
9

Super

eureka

Kidz

Tamil Evaluation Findings - Year 1 & 2 comparison

Children who can read Tamil sentences
Eureka Year 1
100 80 65 % children 60 40 25 20 5 0 7 20 23 0 25 17 33

Children who can read Tamil stories

Eureka Year 1
100 83 80 % Children 60 41 40 35 77 84 75 63 56 57

Eureka Year 2
91 75 95 88 80

Control Year 2
97 92 82 96 95 87

Eureka Year 2
90

Control Year 2
92 87 76

54

Std 1

Std 2

Std 3

Std 4

Std 5

Std 6

Std 2

Std 3

Std 4

Std 5

Std 6

Using the same approach, we see that sentence and story reading abilities are much higher in Eureka villages compared to control, across classes. Usually by Std 2&3, most children learn to read words. By identifying what they already know and helping them read sentences, a large number of them have progressed to the next levels of reading. There is a definitive learning acceleration in Eureka by atleast 1 or 2 classes, in the ability to read sentences and stories. Super
eureka

10

Kidz

Tamil Evaluation Findings Learning acceleration

Children who can read Tamil stories
Story reading skill is an important skill expected by Std 3. Children in control have learnt the skill over several years a few have learnt in Std 3 (only 35%), a few more in Std 4 and so on. A child who learns to read much later than expected, soon falls behind on learning other skills. This delays learning and undermines the childs confidence. Learning delayed is learning deprived! Hence accelerating learning is very important.
100 80 % Children 60 40 20 0 17 Std 2 Std 3 Std 4 Std 5 Std 6 35 56 57 76

Control Year 2

Children who can read Tamil stories

Eureka Year 2 100 80 77 83 Control Year 2 90 92

% Children

60 41 40 20 0 17 Std 2 Std 3 Std 4 Std 5 35

76

56

57

Std 6

11

Super

eureka

Kidz

Tamil Evaluation Findings Using Class Cohorts

Year 1 (2010-11)
Eureka Village Std 1 Std 2 Std 3 Std 4 Std 5

Year 2 (2011-12)
Eureka Village Std 1 Std 2 Std 3 Std 4 Std 5

Class Cohort starting in Std 1 Class Cohort starting in Std 2

Class Cohort starting in Std 3

Class Cohort starting in Std 4 So far we have compared each Stds performance every year Std 1 in baseline, Std 1 in Year 1 and Std 1 in Year 2. Since the evaluation was done in the same set of Eureka and Control villages, in both the years, we can look at how a class of children (Class Cohort) have performed over the two years. Henceforth we will use cohorts to understand the impact of the Eureka program over two years, in a better way.

Control Village Std 1 Std 2

Control Village Std 1 Std 2

Std 3
Std 4 Std 5
12

Std 3
Std 4 Std 5

Super

eureka

Kidz

Tamil Evaluation Findings Using Class Cohorts

Class cohort starting in Std 1: Words
100 80 % children 60 40 20 0 3 0 29 27

Class cohort starting in Std 2: Sentences

90
% children 58 100 80 60 40 20 0 3 3 33 29 54 91

Eureka

Control

Eureka

Control

Class cohort starting in Std 3: Story

100
80 % children 60 40 20 0 11 4 14 56 63

In Std 1 and Std 2, both Eureka & Control cohorts were nearly at the same level in Mar-2011.
83

In the 2nd year, many more children in Eureka cohort (90%) have learnt the key skill compared to control. Control cohort starting in Std 3, has good gains in the 2nd year (ie in Std 4). Many in Eureka cohort (63%) learnt the skill in the 1st year itself.

Eureka

Control

Overall, in Eureka, the specific focus on key skills for lower classes has helped. Super
eureka

13

Kidz

Math Evaluation Findings

The key focus skills in Math are Number recognition 2 digit and 2 digit Addition Subtraction Multiplication Division Additionally, from Std 4 onwards Fractions Measurement Geometry Word problems

14

Super

eureka

Kidz

Math Evaluation Findings

Children who can identify 2 digit numbers
Eureka Year 1 100% 80% 60% 40% 20% 0%
Std 1 Std 2 Std 3 Std 4

Children who can identify 3 digit numbers

Eureka Y1 100% 80% 60% 40% 20% 0% 19% 9% 43% 20% 14% 84% 73% 57% 51% 51% Eureka Y2 Control Y2 93% 96% 86% 87%

Eureka Year 2 99% 94% 78% 88% 88%

82%

Control Year 2 100% 98% 96%

35%

64%

26%

3% Std 1

Std 2

Std 3

Std 4

Std 5

In both 2 and 3 digit numbers, levels in Eureka are much higher than control. 82% Std 1 children in Eureka learnt to recognize 2 digit numbers. Most children in Control are learning the skill only by Std 2.

Let us now use class cohorts to look at various math skills and understand the impact over 2 years.

15

Super

eureka

Kidz

Math Evaluation Findings - Numbers

Class cohort starting in Std 1: 2 digit numbers
100% 80% 60% 40% 20% 0% 3% 3% 26% 16% 78% 94% 100% 80% 60% 40% 20% 0% 16% 18% 57%

Class cohort starting in Std 3: 3 digit numbers

93%

Accelerated learning

51%

21%

Eureka

Control

Eureka

Control

Both Eureka and Control cohorts starting in Std 1, have made good improvements in 2 digit number recognition, by the end of year 2. By the end of 2 years, the control cohort starting in Std 3, still has close to 50% children (compared to 7% in Eureka) who have not learnt to recognize 3 digit numbers. Number recognition is a key skill which helps children move on to higher skills like 2 digit addition, subtraction etc. Identifying children who did not have this skill and helping them learn through simple activities has helped ensure that almost all children learnt. Super
eureka

16

Kidz

Math Evaluation Findings - Addition

Class cohort starting in Std 2: addition We see similar trends
100% in subtraction 100% 72% 61% 32% 13% 3% 29% 80% 60% 40% 20% 0% 15% 7% 57% 74%

Class cohort starting in Std 3: addition

84%

for Std 2 cohort in 80% Eureka & Control. 60%

40% 20% 0%

50%

Eureka

Control

Eureka

Control

In 2 years, a larger proportion of children in Eureka cohort compared to control, have learnt addition (including addition of 2 digit numbers with carry). While there is a good improvement in year 1, in control cohort, there is a drop in learning in year 2! This could mean that no new (or very few additional) children have learnt the skill in control. In Eureka, since most children had learnt to recognize 2 digit numbers in Std 1 & 2, they have progressed to next level skills like addition. Super
eureka

17

Kidz

Math Evaluation Findings - Subtraction

Class cohort starting in Std 3: subtraction
100% 80% 60% 69% 33% 7% 3% 68% 23% 100% 80% 60% 73% 66% 44% 37%

Class cohort starting in Std 4: subtraction

40%
20% 0%

40%
20% 0%

15%
9%

Eureka

Control

Eureka

Control

Although Control and Eureka cohorts started at nearly same low levels, a large proportion of children (around 70%) in Eureka learnt subtraction by first year. The trend of drop in learning levels in year 2, continues both in Eureka and Control. In Eureka too, the levels have either remained same or dropped for Std 4&5 (as shown in Mar2012 for cohorts starting in Std 3&4). But overall, more than 65% children in these classes know to subtract.

18

Super

eureka

Kidz

Math Evaluation Findings - Subtraction

We did a smaller sample assessment in 5 Eureka villages in July 2011, to assess learning retention over summer.
% Children

Class cohort starting in Std 3: Subtraction

100
80 60 60 40 20 1 0 29 49

We observed a drop in reading and math skill levels in different classes, over summer! Eg., 49% children could do subtraction at the end of Std 3 (Mar 2011) but only 29% retained at the beginning of next academic year (July 2011).

A loss of learning over summer could mean, first getting children to relearn their lost skill and also help additional children learn the skill.

The trend of drop in learning is alarming!

There is a need to explore the reasons for this further, and help children sustain what they have learnt. This also indicates the need to explore multiple ways of reinforcing concepts and skills.
eureka

19

Super

Kidz

Math Evaluation Findings Multiplication, Division

Class cohort starting in Std 3: multiplication
100% 80% 60% 40% 20% 11% 0% 5% 42% 17% 59% 100% 80% 60% 40% 56% 64%

Class cohort starting in Std 4: division

8%

20% 0%

17% 4% 3%

26%

Eureka

Control

Eureka

Control

60%-65% children in Eureka Std 3 and 4 cohorts know multiplication and division by the time they are in Std 4 and 5. Ability to add and subtract is important to learn complex skills like multiplication and division. A large proportion of children (around 70%) in Eureka, had learnt addition and subtraction in lower classes (learning acceleration). This has helped them progress to learn the next level skills, when they were taught in classes 3 and above. Super
eureka

20

Kidz

Math Evaluation Findings Std 5&6

Fractions, Shapes, Measurement, Word Problems (Std 5 & 6)
Eureka Y1 100% 80% 60% 40% 20% 14% 0% 36% 3% 2% 25% 39% 70% 38% 43% 10% 10% 63% 42% 22% Eureka Y2 Control Y1 Control Y2

57% 17%

We started focusing on science and higher skills in math for Std 6 and above, as a pilot last year. Hence we are not comparing the cohorts for Std 5 or 6 this year.

Fraction addition

Basic shapes

Area

Word problems

For Std 5&6, in addition to skills like multiplication and division, this year, we focused on higher level skills like Fractions, Shapes, Measurement and word problems. Across the skills, compared to year 1, levels in control have remained more or less same. Eureka levels show good improvements. As more children learn basic skills, we expect these numbers to go up in the next year. Clearly we need to strengthen our focus on fractions and word problems.
eureka

21

Super

Kidz

Math Evaluation Findings Std 7&8

Fractions, Shapes, Measurement, Word Problems (Std 7 & 8)
Eureka-Y2 100% 80% % children 60% 40% 20% 0% Fraction addtition Shapes Perimeter Area Word Problems 45% 79% 50% 70% 34% 70% 36% Control-Y2

We extended Eureka classes for Std 7 & 8 in year 2. Std 7 & 8 children from both Eureka and Control villages were assessed on skills like fractions, geometry, measurement and word problems in the Mar 2012 evaluation.

6%

6%

29%

Consistently, more children in Eureka have learnt the skills compared to control. In areas like word problems and area measurement, we see 70% Std 7&8 children in Eureka know the skill atleast 34% more children than control. Many children who are now in Std 7 or 8, have been part of the Eureka program for the last 1-2 years. We believe that accelerating their learning in the previous year has helped a larger number of children to learn to newer skills thus helping in learning augmentation. Super
eureka

22

Kidz

English Evaluation Findings The key skills in focus in English are

Reading letters Reading small and long words Reading sentences

23

Super

eureka

Kidz

English Evaluation Findings - Vocabulary

Children who have vocabulary (action words)
Eureka Year 1 100 Eureka Year 2 Control Year 2

80
60 40 20 0 30 18 9

65

62 42

68 48

50 36

1
Std 1

4 Std 2

5 Std 3

8 Std 4

Std 5

We can observe significant improvements in Eureka between Year 1 and Year 2. The levels in Eureka in year 2 are much higher than Control. Focus on vocabulary (names of things, actions etc) familiarizes children with the language.

24

Let us now use class cohorts to look at English reading skills.

Super
eureka

Kidz

English Evaluation Findings Letters, Words

Class cohort starting in Std 1: reading letters
Eureka 100 Control 84 50 24 5 100

Class cohort starting in Std 2: reading short words

Eureka Control

80
60 40 20

80
60 40 20 2 13 4 5

58

At the end of year 1, the levels in both Eureka and Control were nearly similar. But in the second year, Eureka levels over control have been substantially higher (34%50%). Eg., 58% children in Std 2 Eureka cohort learnt to read words at the end of 2 years compared to 8% in control. In lower classes, in year 1, the focus was on introductory English which involved a few spoken components. The gains in reading skill were less. But the focus on reading in year 2, brought about significant improvements. Super
eureka

25

Kidz

English Evaluation Findings Words, Sentences

Class cohort starting in Std 3: reading long words
Eureka 100 80 60 40 20 0 Control 100 80 60 40 20 0

Class cohort starting in Std 4: reading sentences

Eureka Control

54 21 1 4 11

57 24 6 1

27

The number of children who could read words and sentences in Std 3&4 Control cohorts has remained low. Close to 55% or more children in Std 3&4 Eureka cohorts have learnt to read words and sentences by the end of 2 years. The introductory spoken English and focus on letters in year 1, has helped children starting in Std 3 &4 in Eureka, move to word and sentence reading levels. We need to focus on approaches to ensure that the remaining 45% children in Eureka also catch up on the skills. Super
eureka

26

Kidz

Pre-primary Evaluation Findings

Language & Cognitive Skills (5 year olds)
Eureka 100 80 % Children 60 40 20 0 Picture description Sequencing shapes Sequencing pictures Identifying fruits Matching pictures Identifying numbers Counting objects Tamil letters English letters 69 53 28 13 8 68 61 Control

63
52 34 23 3 28

21

16

The pre-primary program has both 4 & 5 year olds. In all skills, children in Eureka are doing better than in Control. The levels in skills like identifying numbers, counting, letters are overall low in Eureka indicating the need to strengthen these, so that children are familiar with the shapes and sounds of letters and numbers, when these are introduced in school later.

27

Super

eureka

Kidz

Summary Of Findings
Children in Eureka villages are doing much better than Control in all skills in all subjects (Tamil, Math & English), across standards. There is a clear acceleration of learning by atleast 1-2 standards in Eureka villages across several skills. Performance of children in Eureka villages, in Tamil reading skills, across classes has been very significant.

Year 1 evaluation indicated the need to strengthen our focus on lower classes. The increased focus in Std 1-3 in Eureka villages, has resulted in very good improvement across all key skills in Tamil and Math, in these classes.
In Math, a larger proportion of children in Eureka villages have learnt number recognition in lower classes, and skills like multiplication and division in Std 4 and above. More focus is needed on addition and subtraction skills in Std 4-5 and fractions in Std 5 and above. We observed an alarming trend of drop in math learning levels in control villages (and in skills like addition and subtraction in Std 4-5 in Eureka villages). The reason behind this needs to be explored further. We also need to explore multiple ways to reinforce concepts and skills. We introduced classes for Std 7&8 last year. A larger proportion of children in Eureka villages have learnt key skills in Math compared to control. Many children being part of the Eureka program in the previous year, has also contributed to this gain over control. The focus in year 1 on introductory and spoken English seemed to have resulted in better levels in Eureka. While levels are much higher than control, in higher classes there is a need to help those falling behind to catch up on these skills.

28

Super

eureka

Kidz

Way Forward
We must continue our focus on fundamental skills for lower classes, as this is directly accelerating learning, and helping children move on to learning higher level skills. On analyzing the data, we see that while children acquire skills in a short span of 2-3 months, this learning spike needs to be sustained. There is a need to use multiple approaches to reinforce the concept and the skill, so that children master the newly learnt skill. Our regular classes have clearly led to improvements in a large number of children. However there are still a few children who need additional focus to learn a skill. Last year, in addition to regular classes, we had a team visiting villages, to specifically identify such children (who had not yet acquired a particular skill) and help them improve. This specially trained team gave children individual attention and used multiple teaching approaches to help them learn a particular skill. We will continue to use this approach in the coming years, to ensure that even the last set of children in every village learn.

29

Super

eureka

Kidz