Tempe PRE Year One

Implementation Report June 8, 2018

Contents
Executive Summary......................................................................................1

Introduction ................................................................................................3

Overview of the Study Participants .................................................................4

Year One Baseline Findings ............................................................................6

Conclusion and Next Steps .......................................................................... 15

Appendix .................................................................................................. 17
Executive Summary

Background

High quality preschool sets children on a path that leads to college or career,
particularly for those living in low-income households and English language
learners.1 In response to a study showing that the City of Tempe had some of the
lowest levels of kindergarten readiness in the state of Arizona, the Tempe City
Council invested a total of approximately $6 million into the 2-year Tempe PRE
pilot program, designed to increase access to high quality preschool for high need
3- and 4-year old children in Tempe. This report highlights data from children and
teachers participating in Tempe PRE and describes the progress of Tempe PRE at
the end of the first year of implementation.

Year One Baseline Findings

Data from the first year of the Tempe PRE pilot program were analyzed to
understand the progress that children and teachers have achieved during the first
year of implementation. Results revealed that:

Tempe PRE provider sites entered the program with high quality ratings. At
baseline, Tempe PRE classrooms’ mean quality scores were significantly higher
than Quality First (QF) classrooms’ scores in the classroom environment and
emotional support domains.

 67% of Tempe PRE classrooms (10 classrooms) achieved a three star

rating or higher at baseline, indicating high quality has been achieved at
the majority of Tempe PRE classrooms upon program implementation.

Children showed increases in their developmental levels between baseline

and the end of year one of the Tempe PRE program. Results showed that the
most children made gains in the social emotional and literacy domains, indicating
the positive impact of early childhood programs on these areas.

 Results revealed that younger children (age 3) improved their expected

developmental level in the social-emotional and language domains more
often than older children (age 4 and 5).
 Results revealed that familial income is related to significant improvements
in children’s expected developmental levels in the mathematics domain.

Student dropouts and average rates of attendance were unrelated to

quality star ratings. The average star rating of classrooms that had at least one
child dropout was 2.7.

 The pattern of absenteeism was similar for teachers across all star rating
levels (average of 20-21 absences per year across all classrooms).
 The four-star rated classroom had a lower maximum number of absences,
which will be explored further later in the evaluation.

Teacher trainings, coaching, and professional development are beneficial

1
Yoshiwaka, H., et al., (2013). Investing in Our Future: The Evidence Base on Preschool Education,
Foundation for Child Development.

June 2018 1
Year One Implementation Report Draft Executive Summary

to teachers. Trainings increased teachers’ knowledge of the key components of

the High Scope Curriculum and its implementation.

 Teachers reported plans to incorporate the new skills they learned (i.e.,
participatory learning and conflict resolution techniques) into their daily
practice.
 Teachers reported a need for improved communication between the
institutions participating in Tempe PRE (Tempe PRE staff, school districts,
teachers) to make the program run more smoothly and be more effective
for students.

Evaluation Next Steps

Year one of Tempe PRE program implementation resulted in several improvements

across Tempe PRE sites. Early indicators show that additional support for children
and teachers in Early Childhood Education (ECE) settings results in positive
changes that are likely to set children on a path to success. Sites will continue to
collect data during subsequent years of this pilot program and the evaluation team
will continue to investigate whether the goals of the program are met. With
additional data, the evaluation team will be able to conduct more in-depth analyses
focused on examining student and teacher progress, growth, and development
across time. The evaluation team will collect additional primary data (e.g., student-
teacher interactions, students’ home environment, child-caregiver interactions) to
better understand the underlying factors that impact the outcomes of the program.

June 2018 2
Introduction

Involvement in high quality preschool programs is related to improved academic

and behavioral outcomes for children at kindergarten entry, and can improve
children’s likelihood of attending college, entering the workforce, and avoiding
criminal activity in the long-term.2 3 After a study found that nearly two-thirds of
the City of Tempe’s children were performing below the widely held expectations
for reading and language in kindergarten, the City of Tempe Human Service
department proposed a plan to improve the kindergarten readiness of children in
the city by expanding access to high quality preschool for children in low income
households. To this end, the Tempe City Council invested a total of approximately
$6 million into the implementation of a high quality preschool program, Tempe PRE
(Preschool Resource Expansion). Tempe PRE is a two-year pilot program launched
in 2017 to increase access to high quality preschool at no cost for 3- and 4-year old
children living in Tempe, Arizona who are at or below 200% of the Federal Poverty
Level.

This report describes the progress during Year One of the Tempe PRE pilot
program, addressing the following:

Overview of Study Participants: Who is participating in Tempe PRE?

Classroom Assessments: What is the quality of Tempe PRE classrooms at

baseline?

Child Development: Are children enrolled in the Tempe PRE program meeting
widely held expectations for development after one year in the program?

Dosage and Retention: Do patterns of dropouts and attendance vary by Tempe

PRE providers’ baseline Quality First star rating?

Professional Development: What type of professional development services are

offered and what are teachers’ perspectives/reflections on the professional
development they received?

This report includes a snapshot of the students and teachers engaged in the first
year of implementation of the Tempe PRE pilot program, the progress made at the
end of the first year of program implementation and the next steps for the
evaluation. Findings reported as “significant” are statistically significant. Detailed
information about the data sources used to develop this report and the limitations
of the year one analysis can be found in Appendix A and B, respectively.

2
Takanashi, R. (2004). Reconsidering when education begins. What happens before
kindergarten matters. New York: Foundation for Child Development. Retrieved
at: http://fcd-us.org/sites/default/files/ReconsideringWhenEducationBegins.pdf
3
Campbell, F., Ramey, C. T., Pungello, E., Miller-Johnson, S., & Sparling,
J. J. (2002). Early childhood education: Young adult outcomes from the
Abecedarian Project. Applied developmental Science, 6(1), 42-57.
DOI:10.1207/S1532480XADS0601_05

June 2018 3
Overview of the Study
Participants

Tempe PRE was offered at schools in the Tempe Elementary and Kyrene school
districts during year one of the pilot. Within these two districts, Tempe PRE is being
offered in 15 classrooms to a total of 259 children.4 For the year one evaluation, we
are able to compare the quality of the 15 Tempe PRE classrooms to 753 classrooms
in the City of Tempe implementing Quality First.

As displayed in Exhibit 1, most children served by Tempe PRE are English speaking
(83%), and identify as Spanish/Hispanic/Latino (26%), White (22%), and Black or
African American (18%). Tempe PRE serves a fairly even number of female (55%)
and male (45%) children, ranging from three to five years of age (4.3 years old on
average). Tempe PRE children live in households with an average of four people
and a gross annual income of $26,565.

Exhibit 1. Tempe PRE student demographics

Frequency Percent

Race/Ethnicity

Spanish/Hispanic/Latino5 51 25.5

White 43 21.5

Black or African American 35 17.5

Some Other Race 11 5.5

American Indian or Alaska Native 9 4.5

More than one race 9 4.5
Asian 6 3.0

Native Hawaiian or Other Pacific Islander 1 0.5

Unknown 35 17.5

Gender

Female 110 55.0

Male 90 45.0

Language

English 165 82.5

Spanish 28 14.0

Chinese (all dialects) 2 1.0

Telugu 2 1.0

4
An additional five classrooms began implementation halfway through the school year and are not
included in these analyses.
5
The Spanish/Hispanic/Latino category consists of children indicating their ethnicity as
Spanish/Hispanic/Latino and their race as White. The White category consists of children who
indicated their ethnicity as Not Spanish/Hispanic/Latino and their race as White.

June 2018 4
Year One Implementation Report Draft Overview of the Study Participants

French 1 0.5

Hindi 1 0.5

Nepali 1 0.5

Familial Characteristics Frequency Mean

Annual Gross Income 191 $26,565

Household size 194 4.2

Age 197 4.3

June 2018 5
Year One Baseline
Findings

High quality preschool gives children a strong start on the path that leads to
college or a career, particularly for those living in low-income households and
English language learners.6 There are also economic benefits to high quality
preschool, as research has found that about $7 are generated for every dollar
invested in high quality preschool.7 Year one findings for Tempe PRE’s classroom
assessments, professional development, children’s development and dosage and
retention factors are described below.

Classroom Assessments

Highly trained Quality First assessors with professional experience in early care and
education administer assessments of the classroom environment using the
Environmental Rating Scale (ERS), the Classroom Assessment Scoring System
(CLASS), and the Quality First Points Scale. The ERS measures the overall
classroom environment, the CLASS measures emotional support, classroom
organization, and instructional support, and the Quality First Points Scale measures
staff qualifications, administrative practices, and curriculum and child assessment.
These measures of quality are particularly important as studies show that social-
emotional climate and instructional support are generally strong predictors of
children’s development.8

Tempe PRE providers scored in the high range on emotional support (6.73) and
classroom organization (6.07), but in the low range on classroom environment
(3.40) and instructional support (2.39). This suggests that Tempe PRE teachers
may need additional support in ensuring a positive classroom environment and
providing instructional support to their students. However, low scores in the
instructional support domain are normal, as the instructional support domain has
the lowest average quality score among preschools nationwide.9 Exhibit 2 shows
the average scores in each quality domain for the Tempe PRE and Quality First
providers.

6
Yoshiwaka, H., et al., (2013). Investing in Our Future: The Evidence Base on Preschool Education,
Foundation for Child Development.
7
Elango, S., et al., (2014). Early Childhood Education (working paper #21766, National Bureau of
Economic Research, 2015); Timothy Bartik, From Preschool to Prosperity: The Economic Payoff to
Early Childhood Education.
8
LoCasale-Crouch, J., Konold, T., Pianta, R., Howes, C., Burchinal, M., Bryant, D. & Barbarin, O.
(2007). Observed classroom quality profiles in state-funded pre-kindergarten programs and
associations with teacher, program, and classroom characteristics. Early Childhood Research
Quarterly, 22(1), 3-17.
9
Head Start: Early Childhood Learning & Knowledge Center. (2018). Use of Classrooms Assessment
Scoring System (CLASS) in Head Start. https://eclkc.ohs.acf.hhs.gov/designation-renewal-
system/faq/use-classroom-assessment-scoring-system-classr-head-start

June 2018 6
Year One Implementation Report Draft Year One Baseline Findings

Exhibit 2. Comparison of Tempe PRE and Quality First Providers’

Baseline Quality Scale Scores10 11

Classroom
Environment* 3.1 3.4

Instructional Support 2.2 2.4

Classroom Organization 5.96.1

Emotional Support* 6.3 6.7

1------------------------------------ 4 -------------------------------------------7
Low Mid High

On average, Tempe PRE classrooms scored significantly higher than QF classrooms

in the classroom environment and emotional support domains. The greatest
difference in average quality scores between Tempe PRE and QF classrooms was in
the emotional support domain.

Another aspect of Quality First is the quality star rating (detailed in Exhibit 3),
calculated from classrooms’ ERS, CLASS, and Quality First Points Scale scores. The
highest quality classrooms are expected to achieve a star rating of 3 or higher.

Exhibit 3. Quality First Five Tier Scale

Rising Star Progressing Star Quality Quality Plus Highest Quality

    

Committed to quality Approaching quality Meets quality Exceeds quality Far exceeds quality
improvement standards standards standards standards

At baseline, significantly more Tempe PRE classrooms (67%) received a quality star
rating of three or more than did QF classrooms (31%). In addition, a significantly
greater proportion of Quality First classrooms had a two star rating (62%) than did
Tempe PRE classrooms (33%). A comparison of Tempe PRE and Quality First
providers’ quality star ratings at baseline are displayed in Exhibit 4.

10
Fourteen providers were scored on emotional support, classroom organization, and instructional
support, and 15 providers were scored on the classroom environment.
11
Asterisks indicate significance at the p<.05 level.

June 2018 7
Year One Implementation Report Draft Year One Baseline Findings

Exhibit 4. Baseline Quality Star Ratings12

Tempe PRE
33% 60% 7%
(n=15)

QF sample
8% 62% 19% 9% 2%
(n=753)

Total
8% 61% 20% 9% 2%
(n=768)

1-star 2-star 3-star 4-star 5-star

Future evaluation activities will continue to assess growth and change in the quality
star ratings of Tempe PRE classrooms, including changes in the relationships
between teacher education and experience and their quality star ratings, as more
data becomes available.

Children’s Development during Tempe PRE Program Year One

Preschool can reduce the early achievement gap by preparing children for
kindergarten entry, and can equip children with the social-emotional skills needed
to excel in the first years of school and beyond.13 The long-term benefits of high
quality early learning and development experiences in preschool include increased
high school graduation rates, increased economic well-being, better physical
health, less criminal activity, and the long-term development of a high quality
professional workforce.14

Early childhood research experts highlight the importance of documenting students’

development using well-planned assessments that encompass the whole child
(e.g., academic, physical, social-emotional), such as Teaching Strategies GOLD
(TSG). TSG tracks students’ development in six core areas (mathematics, social-
emotional, physical, language, cognitive, and literacy). Students’ raw scores in
each domain at each time point are scaled, and scaled scores are used to
determine whether each student is below their expected developmental level,
meets their expected developmental level, or exceeds their expected
developmental level. We examined changes in the degree to which students met
widely held developmental expectations in each domain across year one of
participation in Tempe PRE by determining the direction in which each child’s
developmental level changed from the beginning to the end of program year one.
Exhibit 5 shows Tempe PRE students’ developmental levels in the six core domains
at baseline (Fall 2017) and at the end of Tempe PRE year one (Spring 2018).

As anticipated, at preschool entry, Tempe PRE students are typically below or

meeting their expected developmental level in all domains. The developmental
domains in which the most children fell below widely held expectations at preschool
entry are the social emotional (76%), literacy (70%), and cognitive (64%)
domains. At the end of program year one, most children shifted from performing
below their expected developmental level to meeting or exceeding developmental
expectations in all domains. Most prominently, the percentage of children meeting
or exceeding social-emotional and literacy developmental expectations increased
between baseline and the end of year 1, suggesting that social-emotional and
literacy skills may be particularly sensitive to time spent in preschool. The lowest

12
Asterisks indicate significance at the p<.05 level.
13
Child Trends (2018). High-quality preschool can support health development and learning.
14
National Association for the Education of Young Children (NAEYC). (2012). http://www.naeyc.org/

June 2018 8
Year One Implementation Report Draft Year One Baseline Findings

increase in the language domain indicates that language takes longer to develop
and often requires specialist support.

Exhibit 5. TSG domain scores for Tempe PRE students15

Fall 2017 58% 42% 1%

Mathematics (n=160)

Spring 2018 16% 74% 10%

Fall 2017 70% 30%

Literacy (n=159)

Spring 2018 8% 80% 13%

Fall 2017 64% 35% 1%

Cognitive (n=160)

Spring 2018 13% 79% 8%

Fall 2017 60% 40% 1%

Language (n=200)

Spring 2018 21% 69% 11%

Fall 2017 56% 44%

Physical (n=200)

Spring 2018 13% 71% 16%

Fall 2017 76% 23% 2%

Social Emotional (n=200)

Spring 2018 17% 64% 16%

Below Meeting Exceeding

15
Data displayed in Exhibit 5 represents students who completed an entire year of Tempe PRE, thus
had data from the Fall and Spring assessments. Not all students received a score in every TSG
domain.

June 2018 9
Year One Implementation Report Draft Year One Baseline Findings

Exploration of the child and familial factors related to children’s development in

each of the six developmental domains showed that children’s age and familial
income were most often related to changes in children’s developmental levels
across the first year of the Tempe PRE program.

 Younger children (age 3) improved their expected developmental

level in the social-emotional domain more often than older
children (age 4 and 5). It is possible that it is harder to create
social-emotional changes in older children because they have been
immersed in social settings longer.

 Younger children (age 3) also improved their expected

developmental level in the language domain more often than older
children. Language acquisition is emerging rapidly around age
three and slows down at age four to five years old16, meaning that
children that are three years old are expected to have greater
changes in their language skills than children that are slightly
older.

 Familial income is related to significant improvements in children’s

expected developmental levels in the mathematics domain. This
finding is consistent with the vast body of literature that supports
the notion that family income has a significant impact on children’s
academic success.17

As implementation of the Tempe PRE pilot program continues, the evaluation will
further explore children’s development and kindergarten readiness in relation to
child and familial factors, teachers’ involvement in professional development
activities, and parents’ involvement in parenting support programs.

The Relationship between Program Dosage and Retention and

Program Access and Quality

A meta-analysis of 34 preschool programs found that children attending programs

for at least 20 hours per week experienced better developmental outcomes than
children attending a program for less than 20 hours per week. Additionally, children
that attend more years of preschool had higher levels of cognitive development and
academic achievement as compared to children who attended a preschool program
for fewer years.18 Baseline data on Tempe PRE’s retention and dosage is presented
below.

Tempe PRE Student Retention

During the first year of Tempe PRE, a total of 48 children dropped out of the
program or left their program site before the school year ended. Thirteen of the 15
Tempe PRE classrooms had at least one student dropout, with an average of four
dropouts per classroom. Exhibit 6 shows the number of children in each classroom

16
Belsky, J. (2010). Experiencing the lifespan. New York: Worth Publishers.
17
Dahl, G., Lochner, L. (2009). The impact of family income on child achievement: evidence from the
earned income tax credit. Retrieved from:
https://www.researchgate.net/profile/Lance_Lochner/publication/234708497_The_Impact_of_Family
_Income_on_Child_Achievement_Discussion_Paper_No_1305-
05/links/09e4150f44ba4ee899000000.pdf
18
Nelson, G., Westhues, A. and MacLeod, J. (2003). “A meta-analysis of longitudinal research on
preschool prevention programs for children.” Prevention & Treatment. 6 (1). Retrieved from
https://www.researchgate.net/publication/232570779_A_Meta-Analysis_of_
Longitudinal_Research_on_Preschool_Prevention_Programs_for_Children

June 2018 10
Year One Implementation Report Draft Year One Baseline Findings

that dropped out of their Tempe PRE program. The 13 Tempe PRE classrooms with
at least one student absent during program year one are denoted A through M
(along the x-axis).

Exhibit 6. Number of dropouts and star rating by Tempe PRE

classroom during the 2017-18 school year

# of dropouts
Star Rating

A B C D E F G H I J K L M

The average star rating of the classrooms that had at least one child dropout was
2.7, which is higher than the overall average quality star rating of all QF and
Tempe PRE classrooms (2.4). Only one of the four star classrooms had a child
dropout, and the classroom with the most dropouts received a two star rating.

Tempe PRE Program Dosage

Chronic absenteeism (defined in the K-12 system as missing 10% or more of days
in a school year) is associated with lower proficiency in reading and math,
increased likelihood of grade retention, and chronic absenteeism in later grades.19
It has also been shown that students who attend ECE programs regularly benefit
the most from services.20 According to pre-K general guidelines, recurring absence
is defined as 20 non-excused absences in a four month period.21 Under those
parameters, a total of 107 Tempe PRE students (of 256) had recurring absences,
which constitutes 42% of the student population.

Of the 48 students that dropped out of Tempe PRE, 38 students had attendance
data available. On average, these students missed 26 days of school, but three of
the students who dropped out missed 80 or more days during the school year. The
259 students that completed year one of Tempe PRE had an average of 20
absences or unexcused tardies during the 2017-18 school year. Though about a
quarter of Tempe PRE children had zero absences (27%) a quarter (25%) of the
children had 31 or more unexcused absences during the school year.

Patterns of absenteeism were similar for teachers across all star rating levels as
their students averaged 20 to 21 absences per year. This suggests that there is
little relationship between student dropout and average attendance rates and
classroom quality.

Also of interest is the high number of absences in the two and three star rated
classrooms. Though there is only one four star rated classroom, the highest

19
Chang, H.N. & Romero, M. (2008) Present, engaged, and accounted for: The critical importance of
addressing chronic absence in the early grades. New York, NY: National Center for Children in
Poverty. Retrieved from http://www.nccp.org/publications/pdf/text_837.pdf
20
http://schools.nyc.gov/NR/rdonlyres/FCE914AB-0F8B-4F69-8A50-
21EBD4EE50A9/0/Attendance_final.pdf
21
NYC Department of Education: PreK for All. Attendance.
http://schools.nyc.gov/NR/rdonlyres/FCE914AB-0F8B-4F69-8A50-
21EBD4EE50A9/0/Attendance_final.pdf

June 2018 11
Year One Implementation Report Draft Year One Baseline Findings

number of absences in that classroom are less than half of the highest number of
absences in the two and three star rated classrooms. Exhibit 7 shows the pattern of
student absences by teachers’ quality star rating.

Exhibit 7. Absences by star ratings

Absences
Minimu Number of
Star Rating Mean Median Maximum
m teachers
Two Stars 20 14 0 88 5

Three Stars 21 17 0 86 9

Four Stars 20 19 1 40 1
Total 20 16 0 88 15

During the next stage, the evaluation will examine relationships between quality
star ratings and program dosage and retention over time, as well as how dosage
and retention interacts with the effects of participating in the Tempe PRE program
on children’s developmental levels during pre-K and kindergarten readiness levels
at kindergarten entry.

Teacher Curriculum and Assessment Training, Coaching, and

Professional Development

Teachers in the Tempe PRE program participate in trainings, receive one-on-one

coaching, and engage in professional development. The knowledge, skills, and
practices of early childhood educators are important factors in determining how
prepared children are for kindergarten. Sustained professional development (PD)
for teachers has been associated with more positive and stimulating teacher
behavior, helps teachers meet the demands of diverse student needs, improves
teacher engagement with parents, and allows teachers to develop an active agenda
for their own professional growth, all of which has been linked to positive child
outcomes.22 In addition, one-on-one teacher coaching has been shown to be
integral to ECE programs and has been linked to high quality student-teacher
interactions and less teacher burnout and turnover.23 24

Teacher Trainings

On average, teachers attended a total of nine training sessions during the first year
of Tempe PRE. Trainings offered included the foundational High Scope Curriculum
training; specialized High Scope trainings focused on interaction strategies,
problem solving, and becoming an intentional teacher; TSG training; social-
emotional development-focused Arizona Early Learning Standards training; and
language and literature-focused Arizona Early Learning Standards training.

Increased Teacher Knowledge

Teachers who participated in the High Scope Curriculum training reported that

22
Continued Professional Development Framework for Early Childhood Educators:
https://www.childcarelink.gov.sg/ccls/uploads/CPD_Guide_5_FA.pdf
23
Aikens and Akers, Background Review of Existing Literature on Coaching.
24
Kimberly Boller et al., Seeds to Success Modified Field Test: Findings from the Outcomes and
Implementation Studies (Princeton, NJ: Mathematica Policy Research, 2010).

June 2018 12
Year One Implementation Report Draft Year One Baseline Findings

 It increased their knowledge of how to implement the curriculum and its

key components;

 They planned to incorporate participatory learning and conflict

resolution techniques in their daily practice; and
Tempe PRE Professional Development
 They gained a better understanding of key developmental indicators
Opportunities*
and intentional teaching strategies as a result of the training.

Teacher Coaching During Tempe PRE Program Year One  Training on implementation of the
High Scope curriculum and student
Tempe PRE provides teachers with the opportunity to receive coaching support data collection using Teaching
for student behavioral issues, schedule management, classroom management, Strategies GOLD (TSG).
and optimizing the classroom environment. Coaching is voluntary for teachers,
but was utilized by more than half of teachers (8 of 15) during the first year of  Four hours of onsite coaching per
the Tempe PRE program. month, with the goal of identifying
and setting goals for quality
Each coaching session begins with an observation of classroom organization and improvement. The QF coaching
functioning. Following the classroom observation, the coach provides model includes individualized
recommendations to the teacher and feedback about observed strengths and guidance and support, targeted
weaknesses. The recommendations provided by coaches are typically around training and technical assistance,
schedule optimization and improving the quality of the classroom environment, and support for goal development
including tips to ensure students are engaged at all phases of the school day, and implementation.
and ideas for age-appropriate activities.
 Professional development
Teacher Reflections on the First Year of the Tempe PRE Program opportunities to enhance core
competencies and standards, build
Observed Student Social and Behavioral Growth content knowledge, expand their
education and career pathways, and
At the end of the first year of Tempe PRE, teachers were asked about the network with other ECE providers.
successes and challenges they experienced with the program and the areas in
which they believed they would benefit from additional support. Teachers *Teachers’ experiences with trainings, coaching, and
PD are captured in surveys and trainer logs/notes
reported many successes among their students, particularly with their
following participation in these activities.
developmental progress. For example, one teacher reported that her students
“…are showing an interest in coming to school and are excited to be in the
classroom”, and another teacher reported seeing “HUGE social and behavioral
growth” among students.

Need for Improved Communication

Teachers reported that their primary areas of concern during their first year of
Tempe PRE are the qualifications of their instructional assistants and
communication between the institutions participating in Tempe PRE. Specifically,
teachers cited a need for more coordination and clear lines of communication
between teachers, Tempe PRE staff, and school districts. When surveyed about the
degree to which they feel supported by the Tempe PRE staff and the school district,
teachers reported feeling supported by Tempe PRE staff and feeling comfortable
contacting Tempe PRE staff with questions and concerns. They also reported that
Tempe PRE staff helps find solutions to the issues they report. However, fewer
teachers felt supported by their school district, though they still reported feeling
comfortable reaching out to school district staff about their questions and issues
when needed. As this was the first year of Tempe PRE implementation, it is
expected that clear communication pathways will continue to be established and
teachers will feel more supported by both Tempe PRE staff and their school district
as the program develops.

Additional Support and Professional Development

Finally, teachers reported that they would benefit from more support with data

June 2018 13
Year One Implementation Report Draft Year One Baseline Findings

collection, additional training on the High Scope Curriculum, and training for their
instructional aides, and indicated these would be useful topic areas for additional
professional development.

Year two of the Tempe PRE pilot is likely to reveal more about teachers’
experiences with trainings and professional development opportunities associated
with participation in Tempe PRE. The next round of post-teacher training data
collection will allow the linking of teachers’ training and professional development
experiences to other factors to understand how training on these topics is related
to teacher competence and confidence implementing the curriculum and TSG and
quality star ratings.

June 2018 14
Conclusion and Next Steps

Summary of Year One Findings

Data from the first year of the Tempe PRE pilot program were analyzed to
understand the progress that children and teachers have achieved during the first
year of implementation. Results revealed that:

Tempe PRE classrooms showed high quality ratings. At baseline,

Tempe PRE classrooms’ mean quality scores were statistically significantly
higher than QF classrooms’ scores in the classroom environment and
emotional support domains. Additionally, 67% of Tempe PRE classes (10
classrooms) achieved a three star rating or higher, indicating the majority
of Tempe PRE classrooms are high quality upon Tempe PRE program
implementation.

Children showed increases in their developmental levels between

baseline and the end of year one of the Tempe PRE program.
Younger children (age 3) improved their expected developmental level in
the social-emotional and language domains more often than older children
(age 4 and 5). Results also showed that familial income is related to
significant improvements in children’s expected developmental levels in the
mathematics domain.25

Student dropout rates and average rates of attendance were

unrelated to quality star ratings. The average star rating of the classes
that had at least one child dropout was 2.7. In addition, it was found that
the average absenteeism rate was similar for teachers across all star rating
levels (average 20-21 absences per year).

Teacher trainings, coaching, and professional development are

beneficial to teachers. Trainings increased teachers’ knowledge of the
key components of the High Scope Curriculum and how to implement it.
Teachers reported a need for improved communication between the
institutions participating in Tempe PRE (Tempe PRE staff, school districts,
teachers) to make the program run more smoothly and be more effective
for students.

Next Steps of the Evaluation

Sites will continue to collect data during subsequent years of this pilot program,
and the evaluation team will continue to investigate whether the goals of the
program are met. With additional data, the evaluation team will be able to conduct
more in-depth analyses focused on examining student and teacher progress,
growth, and development across time. Specifically, our next steps include:

25
Dahl, G., Lochner, L. (2009). The impact of family income on child achievement: evidence from
the earned income tax credit. Retrieved from:
https://www.researchgate.net/profile/Lance_Lochner/publication/234708497_The_Impact_of_Family
_Income_on_Child_Achievement_Discussion_Paper_No_1305-
05/links/09e4150f44ba4ee899000000.pdf

June 2018 15
Year One Implementation Report Draft Conclusion and Next Steps

1. Examine classroom quality outcomes (e.g., quality star ratings) over time.

2. Link information about teachers’ experiences with trainings, coaching, and

professional development opportunities to other factors (e.g., quality star
rating, competence and confidence using TSG, children’s developmental
levels and kindergarten readiness), and explore whether those experiences
change over time.

3. Assess program implementation fidelity and data collection consistency

and reliability.

4. Use high-level analyses to explore the relationship between participation in

the Tempe PRE program and children’s developmental levels and
kindergarten readiness over time.

5. Examine potential variation in the relationships between quality star

ratings and program dosage and retention over time.

6. Explore whether dosage and retention interacts with the effects of

participating in the Tempe PRE program on children’s developmental levels
during pre-K and kindergarten readiness levels at kindergarten entry.

We will also collect additional primary data to allow us to develop a more in-depth
understanding of each of the three goals. For example, we plan to collect additional
data from teachers to understand teacher confidence and competence in
implementing the High Scope curriculum and student-teacher interactions. We plan
to collect data from caregivers to get a better understanding of children’s home
environment and caregiver interactions and relationships with their preschool-age
children.

June 2018 16
Appendix

Appendix A. Brief description of each data source

Administrative data collected by the Tempe PRE and Quality First programs covers ECE professionals’
education, experience, and professional development participation, and information about providers,
classrooms, and programs.

The Environment Rating Scale (ERS) is an observational assessment that measures quality in ECE
settings by observing various interactions and experiences that are indicators of protection of children’s
health and safety, building positive relationships, and opportunities for stimulation and learning from
experience. The ERS is one of the Quality First program components supporting quality improvement for
ECE providers.

The Classroom Assessment Scoring System (CLASS) is a quality assessment tool used to rate teacher-
child interactions across three domains: Emotional Support, Classroom Organization, and Instructional
Support. Each CLASS domain is scored on a scale of 1 to 7, with research suggesting that Instructional
Support scores of 3 or above, and Emotional Support and Classroom Support scores of 5 or above, lead to
greater gains for children. The CLASS is one of the Quality First program components supporting quality
improvement for ECE providers.

The Quality First Points Scale measures three areas of quality: 1) staff qualifications, 2) administrative
practices, including staff/child ratios, group sizes, and staff retention, and 3) curriculum and child
assessment.

Teaching Strategies GOLD (TSG) is an ongoing observational system for assessing children from birth
through kindergarten. It helps teachers to observe children in the context of every day experiences, which
is an effective way to learn what they know and can do. Teaching Strategies GOLD is based on 38
objectives for development and learning that include predictors of school success and are based on school
readiness standards. The objectives are aligned with the Common Core State Standards, state early
learning guidelines, and the Head Start Child Development and Early Learning Framework.

June 2018 17
Appendix B. Limitations of the Year One Analysis

Though there is a great deal of rich data available about the students, teachers,
and sites that participated in the first year of the Tempe PRE program, and several
meaningful analyses were able to be conducted, there were some limitations to
developing this report.

Ability to match Tempe PRE participants across data sources. The data used
to examine associations among preschool children, their teachers, and their sites
often came from different sources, meaning that each individual’s data needed to
be matched across data sets before being meaningfully analyzed. In some
instances, student, teacher, or site identifiers were not available and participants
had to be matched by name or other identifying feature.

Teaching Strategies GOLD scoring system. The Teaching Strategies GOLD

background documentation reports that children’s scores in each developmental
domain are scaled from zero to 1,000, with a mean of 500, but the raw data must
be scaled using a methodology not described within the documentation. Thus, this
first stage of the evaluation did not report on children’s raw or scaled scores in
each domain, but instead reported on whether children were below, met, or
exceeded widely held expectations for developmental in each domain. If possible,
future analyses will include examination of raw or scaled scores.

Attendance standards. Generally speaking, chronic absenteeism is defined as

missing more than 10% of school days in a given school year. However, as our
team does not have access to the guidelines for absentee levels in preschools in
the state or school district, or the baseline number of children in each classroom to
calculate chronic absenteeism based on the general guideline, we operationalized
chronic absenteeism as missing more than 20 school days in a school year.26

Small sample size. When data is examined in subgroups, the small sample size in
some subgroups limits the complexity of the analyses we are able to conduct due
to decreases in analytic power. For example, analyses run by child age group are
tenuous as there are only 13 children in the three year old age group as compared
to 104 children in the four year old group and 80 children in the five year old
group.

26
NYC Department of Education: Pre-K for All. http://schools.nyc.gov/NR/rdonlyres/FCE914AB-0F8B-
4F69-8A50-21EBD4EE50A9/0/Attendance_final.pdf
harderco.com

Harder+Company Community Research works

with public- and social-sector organizations across
the United States to learn about their impact and
sharpen their strategies to advance social change.
Since 1986, our data-driven, culturally-responsive
approach has helped hundreds of organizations
contribute to positive social impact for vulnerable
communities. Learn more at www.harderco.com.
Follow us on Twitter: @harderco.