Group Discussions on RUPAS Case Study

C:
RUPAS, or the Rural Parents Support Program, was founded by the organization Family Matters. Its
goal was to provide training and resources to parents who lived in rural areas so they could best support
their children. Amy Wilson, the organizations founder, desperately needs more funding to keep the
program going, so an evaluation must be completed. I believe that an internal evaluators would be the
best option in this scenario because they would have the programs best interest at heart. They would also
have a great deal of knowledge about the program already and would be able to research possible ways to
improve it. However, to avoid any possible claims of biases, I think that an evaluator advisor would also
be a smart option to utilize along with the internal evaluator. The internal evaluators would be able to
seek guidance from the advisor and ask questions along the way as well.

D:
The primary stakeholders for RUPAS would be the families that they serve along with Mary Wilson and
Family Matters. The Parent Leaders and the other participating parents would be able to provide
important information based off of their first-hand experiences with the program. The state of
Washington and the organization Childrens Trust would also be stakeholders because they helped fund
RUPAS. The views of the teachers, administrators, and counselors that educate these children would be
extremely important to have as well. How well or how poorly these kids perform in school, both
academically and socially/emotionally, have a direct link to the effectiveness of RUPAS. The views of
church leaders and police departments in that area would also be beneficial to acquire in order to
understand the behavior of these children outside of their school and home.

E:
If I were an external evaluator for RUPAS, I feel that I have numerous qualities and experiences which
would make me a credible choice. I have been an educator for nine years, and I am a leader within my
school. I am part of several committees, a team leader in sixth grade, and the event coordinator for the
school. I have my Masters degree in middle grades education and am working toward a Specialist
degree in instructional technology. I have never performed an official evaluation of a program, but I
constantly have to evaluate my students performance within my classroom. I have also participated in
many peer observations where I had to evaluate my coworkers lessons. Lastly, I am not only evaluated
multiple times per school year by my administrators, but I also have to evaluate myself as well.

I feel I do well communicating with my students, their parents, and my co-workers. I communicate with
parents by email, phone calls, grade sheets sent home, and positive letters about how well their students
are performing/behaving in class. I try to stick to my own rule that for every one negative contact I have
to make with a parent, I will make two positive contacts. This also helps build positive relationships with
my students, which ensures there will be open communication between myself and them. I feel that I am
an advocate for my students and treat them like they are my own with tough love and encouragement.

I consider my personal demeanor a strength due to my positive, easy-going attitude. I work hard to meet
expectations and obtain goals, but I try to remain level-headed and not put too much pressure on myself
or others. I work well with others, and I am a leader but not an overbearing, bossy one. I listen to others
concerns and opinions, and I try to always remain open-minded. As an evaluator, I would not jump to
conclusions, be close-minded, or be too harsh when assessing a program.

F:
When considering the organizational, social, and political factors of RUPAS, the one that immediately
stand out are the families this program serves. These people live in rural areas and are mostly low-
income caucasian families. Another factor is the little funding RUPAS receives, which has caused a need
for evaluation.

G:
I have learned that the RUPAS program is one that attempts to educate and provide resources to parents of
families who live in rural settings and are mostly low-income. Mary Wilson and the founders saw a need
to stop the cycle of parents having children in poverty and those children growing up and having children
in poverty, and so on. It is admirable that RUPAS has a passion to help these families in need because if
they hadnt helped, who would have? This program requires workers/trainees to educate the parent
leaders, and those parent leaders teach and give resources to the other parents in the community. If any of
these people were not willing to be a part of RUPAS, the program would be nonexistent. RUPAS is not a
homeless shelter nor does it provide childcare services. Its goal is to simply educate parents in order to
break the cycle of poverty in this community.

H:
The book stated that RUPAS creates booklets of information for their families along with nominates and
trains the parent leaders. In order to strengthen the program, RUPAS could also create a website along
with social media pages. These would not only spread the word and promote the program, but they could
also be places where the community could easily access information and resources.

I:
Our group discussed many of Alkins points, specifically A program may have many goals, or
outcomes, but which ones are we to look at (2011)? Through this case study we gained clarity on what
kinds of evaluation questions an evaluator should ask. Considering the case study of the non-profit
organization, Rural Parents Support Program (RUPAS), we initially had several different views on who
the primary stakeholders were. We all agreed that Program Administrator, Program Staff Members of
FM, parent leaders, parents, school district leaders, teachers, students and the community are all
stakeholders. We all concluded after further readings and research we gleamed that the individuals listed
above are all indeed Stakeholders but not necessarily the Primary Stakeholders. Those individuals that
should be considered as the Primary Stakeholders by the evaluator should be a more manageable group.
Alkins, 2011, states that the primary stakeholders are the individuals who will be proposing the potential
evaluation questions. Determining specific and meaningful questions is key for the evaluator. The
purpose is not for the stakeholders to already know the answers then there will be no need to ask the
questions. Questions must be relevant and valuable.
Alkins discussed the importance of asking specific questions and not ones like how are we
doing? For program improvement purposes, questions need to be meaningful and useful. They also
need to be answerable and definable. For instance, is the program doing well? is too vague because the
exact definition of doing well is unknown. The primary stakeholders should ask questions which
provide useful answers to make the program more effective and more worthy of receiving funding.
Questions Primary Stakeholders could ask are; What could be done better? Is RUPAS benefiting the
families and the students if so, how? The questions primary stakeholders can ask to strengthen their
funding are to what extent does RUPAS affect systemic education practice? To what extent does RUPAS
change the educational or behavioral policy inside the school system? To what extent has RUPAS
sustained and improved practices that enhance social and academic outcomes for students?

J:
The world of a program is filled with data and with questions. Our group understands the
importance of evaluation questions being clear and relevant. The questions should lead to the DATA
needs. In the case study of RUPAS, the program participants are the Family Matter (FM) Staff, Parent
Leaders, Parents and student the program participants of RUPAS. The Case Study exercises asks us if
when developing questions to gain additional understanding of the data do we pose questions to the
Parent Leaders/Program participants, staff or both. We discussed collecting data that's a representation of
all participants. We are definitely not saying we collect information from ALL participants, but we do
believe that there should be a good mixed representation to prevent bias. Other stakeholders who can
provide us DATA would be school leaders, teachers, the school district, community organizations; youth
based extracurricular personnel, law enforcement, and government agencies. Data from the identified
stakeholders would definitely be direct and indirect. Some possible sources are trend DATA from
RUPAS, past and present data from the school system, community organizations and data.
The program participants in the RUPAS case would be the family leaders and other families receiving
RUPAS support. The questions posed in section one would be for the parent leaders/participants as well
as the RUPAS staff.

For whom or where are data acquired?

Participants: Parent leaders, parents, children

Staff: All RUPAS staff, parent leaders

Other Stakeholders: Education professionals, law enforcement, local youth based

extracurricular leaders

Documents: Archived data from RUPAS, archived data from school system, archived data from
law enforcement

Observation: Program observer, relevant data collected during evaluation

K:
To measure what results came out of the program the following instruments can be used for
obtaining the data. Observation, self-reporting, achievement, questionnaires and existing data are all
quantitative measures that can be used in the evaluation of the program. The population in which they
were serving was rural families, with little to no education, limited transportation. The participants were
in need of resources, training and assistance. Some of the questions that we developed are as follows:
 To what extent will RUPAS impact student performance on standardized testing?
To what extent will RUPAS impact student participants behaviors?
Did student participants attendance improve in school?
All of the questions above could employ the use of a survey, while only a few can be an observation
protocol. The text asks us whether achievement testing could be possible with the children in RUPAS
program? Because of the poor attendance of these students, achievement data may not provide valuable
data for all stakeholders in the program as it may be an inaccurate depiction of all student participants.

L:

Observations:

Pros: Objective Flexible, provides context, in-depth information

Cons: Training required, time consuming, expensive, subjective, a lack of control on the direction of the
participants being observed which can lead to time loss

Interviews:

Pros: Provides in depth information, allows for follow up questions, participants can help each other
answer or elaborate on questions,

Cons: Can digress into irrelevant discussion, depending on the type of interview can lead to unreliable
group think data, limited number of questions, sometimes requires multiple interviewers, skilled
interviewer is required, a lack of control on the direction of the participants being interviewed which can
lead to time loss, reliant on people to complete and return surveys.

Surveys and Questionnaires: Most appropriate for gathering data

Pros: Quick to deploy and analyze, relatively inexpensive, easy to disaggregate and analyze

Cons: Less in-depth responses, inability for follow-up questions or elaboration of key ideas,

M:

When evaluating the effectiveness of the RUPAS program, the evaluators will run into a number
geographical issues that would arise during the evaluation process. The first issue would arise when trying
to gain access to relevant data. When working with an organization such as RUPAS that has programs
spread out among so many communities it is hard to establish and maintain the kinds of relationships that
will produce useful data. Another problem might be collecting relevant data through qualitative and
quantitative means. Useful qualitative data would hard to come by, interviews and focus groups would be
very difficult to conduct with so many stakeholders who would be spread across a large land area. In
terms of observations, how much time will be spent on travel by the observer? How will the observers
travel expenses be paid for? When conducting interviews you have the same sort of issues. Will you pay
interviewees for their travel expenses? How much time and money will be dedicated to letting people
know where the interview site is located? The quality of data might also be adversely affected by the
geographical constraints because the geographical constraints lead to timing constraints. When collecting
data, there are optimal times to do so at a business, school, or with various stakeholders. However, when
travel time is factored into the collecting of data, the evaluator might now always be able to gather data at
the optimal times which could lead to a number of data quality issues. Maybe timing does not allow you
to interview specific people, or timing prohibits you from observing certain individuals working on
certain tasks, or maybe it means that specific interviews or observations just cannot take place because of
timing conflicts. The geographical challenges also lead to cultural challenges. As a program like RUPAS
grows and stretches its reach it encounters new communities and is even influenced by the culture of
these new communities. Some of these challenges might include language barriers, conflict of beliefs and
misunderstanding of that regions cultural norms. This can be challenging to evaluators because different
versions of the same program can have different expectations and viewpoints as to the role of the program
in their community and their willingness to work with you as an evaluator. It would be ideal for the
evaluator if all of these factors remained constant but they dont and it is up to the evaluator to overcome
these challenges.

N:
The three questions we asked of the RUPAS program included the following: To what extent will
RUPAS have on school discipline, attendance and achievement on standardized test scores? We believe
that these questions meet most of the qualifiers for evaluable questions. Based on the information from
the readings that we have about RUPAS, this is an organization that is a few years into a successful
operation with some recent shift in leadership but not really in objectives. It would also seem that RUPAS
would have the appropriate resources to answer our three questions in a relatively short amount of time
and at a reasonable cost. The questions being asked are of tremendous worth; finding reliable answers to
these questions could justify additional financial support. The great thing about the questions is that each
of them would be measurable, and the data collected could be compared to previous data to hopefully
show positive impacts from implementing RUPAS. Time would not be too much of a limiting factor for
the answering of these questions as the data should be collected and compared in a relatively short
amount of time. Gatekeepers of the data being gathered should have no concerns over the rights of
individuals, conflict of interest, or the potential for misleading information because all of these questions
can be answered with simple quantitative data. If more qualitative data collection were to take place then
some of these issues could become a concern. Finally, it is important to consider any potential impact that
politics might have on the answering of these questions and the subsequent impact that might have on the
organization. This possibility of politics entering the evaluation could impact the allocation of money and
possible changes to local education.

O:
Process Variables and Potential Issues (RUPAS Evaluation):

Personnel Large number of personnel who are also spread out across a large geographic area.

Materials With a program with as many objectives as RUPAS keeping track of pre-existing materials,
arrival of new materials and cost of all materials would be a challenge.
Clients Large number of clients who are also spread out across a large geographic area.

Organizational Structure An organization with so many objectives across such a large geographic area
would be hard to establish a chain of command, identify organizational norms and organizational goals.
There are also growing concerns about securing funding for the future existence of RUPAS.

Program Mechanisms Would be encountered when an evaluation actually took place.

P:
To what extent will RUPAS impact school discipline?

What data will be needed?

1. Surveys
2. Number of discipline incidents prior to RUPAS compared to during RUPAS

Who will the data be obtained from?

1. Teachers & principals
2. Behavior records

When will data be acquired?

1. At the start of the school year discipline data will be obtained from previous years.
2. Discipline data obtained at the end of the year.
3. Surveys conducted at the beginning & end of the school year

How will these data be analyzed?

1. Compare prior number of discipline incidents to the current number of incidents.
2. Study the results of the surveys & looks for any areas of possible improvement along with
areas of strength.

To what extent will RUPAS impact the total number of student absences?

What data will be needed?

1. Attendance numbers from previous school years
2. Surveys completed by students, parents, & teachers

Who will the data be obtained from?

1. Attendance records
2. Students, parents, & teachers

When will data be acquired?

1. Data obtained at the beginning of the school year (for numbers from previous years) along
with the end of the year.
2. Beginning & end of the year surveys.

How will these data be analyzed?

 1. Compare the attendance numbers prior to RUPAS & after a full school year using the
program.
2. Study survey results to look for areas to improve along with areas of strength

To what extent will RUPAS impact student performance on standardized testing?

What data will be needed?

1. Standardized tests scores from previous years compared to the year that RUPAS is implemented

Who will the data be obtained from?

1. School standardized testing records

When will data be acquired?

1. Beginining of the year (to look at previous school years) along with the end of the school
year

How will these data be analyzed?

1. Compare prior standardized testing data to end of school year standardized testing data.
1. What does the program do well? What could be done better?
What data are needed?
Attitude measure of certain programs.
Who will data be obtained from?
Sampling of participants in the program
When will data be acquired?
Middle and towards the end of each program.
How will these data be analyzed?
Summary of the data will be conducted.
2. Is RUPAS benefiting the families and the students if so, how?
What data are needed?
Attitude measure of certain programs.
Who will data be obtained from?
Sampling of participants in the program
When will data be acquired?
During the middle and end of session
How will these data be analyzed?
Summary of the data group will be given.
3. To what extent does RUPAS affect systemic education practice?
What data are needed?
Teachers attitudes toward specific group of students regarding behavior and knowledge
Who will data be obtained from?
Sampling of teachers from the district (kindergarten teachers)
When will data be acquired?
Middle and end of the school year
How will these data be analyzed?
Summary of the reports will be given.
4. To what extent does RUPAS change the educational or behavioral policy inside the school system?
What data are needed?
Grades from the districts from which the participants belong
Who will data be obtained from?
Teachers from the district
When will data be acquired?
With parent permission, grades from the schools.
How will these data be analyzed?
Summary of data group. Examination of individual items
5. To what extent has RUPAS sustained and improved practices that enhance social and academic
outcomes for students?
What data are needed?
Random sampling of student progress and final report carts.
Who will data be obtained from?
Parents
When will data be acquired?
Every 9 weeks
How will these data be analyzed?
A Summary of grades will be provided

Q: Potential procedural issues for the RUPAS evaluation:

The evaluation process is always a long and tedious one. In order to receive the necessary feedback
regarding the program the cooperation of parent and staff (teachers) of the district is pertinent. Parents
will have to submit paperwork and be willing to talk to the staff members of RUPAS on a consistent basis
regarding their feelings. Therefore, the parents have to stay in the program long enough for the
evaluations to be effective. Teachers and districts will have to cooperate with the organization in order for
feedback to be relevant. The district is the important component because they are the ones who will be
able to see and measure growth. In summary, meaningful relationships have to be created with the data
keepers along with cost for funding and evaluation standards must be set.

R:
The first question that our group asked in section P that is quantitative is To what extent will RUPAS
impact school discipline? The dependent variable would be the actual number of office referrals
submitted on student participants during the entire school year. The next question is To what extent will
RUPAS impact the total number of student absences? The dependent variable would be the number of
student participant absences during the school year. The final quantitative question is To what extent will
RUPAS impact student performance on standardized testing? The dependent variable for this question
would be the student participants standardized test scores from previous years compared to the year they
began receiving support from RUPAS. The independent variable for all three of these questions would be
the amount of interventions and support RUPAS provided to these participants.
S:
The only portion of one of our questions which is qualitative is the teacher and principal surveys filled out
concerning the student participants behavior and any disciplinary issues that occured. Assuming I
analyzed the data from our questions, I believe I would discover the theme of students participating in the
RUPAS program have a decreased number of office referrals, fewer absences from school, and an
increase in standardized test scores.

T:
Question 1: To what extent will RUPAS impact school discipline?
Stakeholder guidelines:
-What steps will RUPAS take to improve student behavior at school?
- What resources will need to be devoted to this goal?
Question 2: To what extent will RUPAS impact the total number of student absences?
Stakeholder guidelines:
-What impact will increased school attendance have on the student and family?
-What impact will increased school attendance have on the community?
-What resources will need to be devoted to this goal?
Question 3: To what extent will RUPAS impact student performance on standardized testing?
Stakeholder guidelines:
-What steps will be taken to achieve this goal?
-What impact will the attainment of this goal have on school funding?
-What resources will need to be devoted to this goal?

U:

It is imperative for the evaluator(s) and the stakeholders to communicate effectively throughout the
evaluation of the program. The final evaluation report should be sent to the head of the RUPAS
organization, the administrators of the participating schools, and the parents participating in the program.
The head of the RUPAS organization should share it with all of the people who work for the
organization, and the administrators should share the findings with their teachers. I believe that multiple
reports would be necessary due to some of the participant families speaking different languages. I also
think that a more precise and easier to understand version would be beneficial to send to the families
because they most likely will not understand or need to know every single detail of the full formal written
evaluation.

V:

I believe that all of the stakeholders would use the RUPAS findings to discover the strengths and
weaknesses of the program. They would use the findings of any weak areas to help improve the program
and make it more effective. I also think that any other programs who have a similar goal of reaching
various types of families and students would benefit from these findings. They could learn from what
RUPAS does well and maybe even some tips of what not to do.
If Amy Wilson was a user with no previous experience with evaluation reports, I would outline a
decision-making process and list the specific steps sequentially, the data that is available, and the specific
decisions that could be made. I would also train her to use data by giving her information and data about a
similar program to show her how the process works.

W:

The three types of skills needed to carry out this evaluation include technical, social, and cultural. Staff
will need to be able to write the proposal, obtain the contract or agreement, develop the budget, and
manage operations. If the evaluator does not feel that his or her staff is adequately prepared in all of these
areas, further training would be necessary. Funding this type of staff who is fully knowledgeable and
prepared to carry out the evaluation would most likely be an expensive task. The factor that I believe
would have caused difficulty in RUPAS adhering to its timeline would be delays in communicating since
the family participants in the program lived in widespread rural areas.