Tay Pss-Fss Evaluation Toolkit Draft 4-21-16

TRANSITION AGE YOUTH (TAY)
PEER AND FAMILY SUPPORT SERVICES

PROGRAM EVALUATION TOOLKIT

Step-by-Step Technical Assistance Guide:
TAY Peer and Family Support Services: Process and Outcomes
Evaluation

Using the TAY Peer and Family Support Services Toolkit
1. Read the entire Toolkit Document
To familiarize yourself with the project
2. Complete the Feasibility Analysis Checklist (Pages 10-18)

To assess if and how you should conduct the evaluation
3. Use the Step-By-Step Evaluation Guide and Planning Tool (Pages 19-36)
To develop and implement your own evaluation plan
4. Use the External Resources section (Pages 57-58)

To access additional information on how to plan the evaluation, train data collectors, and enter, analyze, and
interpret the data.
5. Use the Evaluation Surveys (Appendix, pages 59-83)

To collect data for the evaluation
6. Use the Double Data Entry Form (Supplementary Document)

To enter the data from the surveys for analysis
7. Use the Report Template (Pages 37-55)

To report and interpret the findings from the evaluation
T A Y P SS F S S E V A L U A T I O N TO O LKI T
Toolkit Authors and Affiliations

Sarah Hiller, MPIA
Steve Tally, PhD
Victoria Ojeda, PhD MPH
Todd Gilmer, PhD

University of California, San Diego (UCSD) School of Medicine, Department of Family Medicine and
Public Health, and the UCSD Health Services Research Center (HSRC).

Acknowledgements
The concept of this Evaluation Toolkit is based on an evaluation originally developed by HSRC for the
County of San Diego Behavioral Health Services. Please note that the original evaluation was not
designed to be TAY-specific, and has been adapted to focus on TAY.

The development of this toolkit was funded by Californias Mental Health Services Oversight and
Accountability Commission (Contract #13MHSOAC014), Sacramento CA.

Table of Contents
GLOSSARY/ACRONYMS ..................................................................................................................... 5
INTRODUCTION .................................................................................................................................... 6
What is the aim of this toolkit, and who should use it? .............................................................................. 6
When is it appropriate to use this toolkit? .................................................................................................. 6
Why was this toolkit developed? ............................................................................................................... 6
What does this toolkit include? .................................................................................................................. 8
1. Feasibility Analysis Checklist ............................................................................................................ 8
2. Step-by-Step Evaluation Guide and Planning Tool ........................................................................... 8
3. External Resources ........................................................................................................................... 8
4. Evaluation Survey Measures ............................................................................................................. 8
5. Double Data Entry Form Template .................................................................................................... 8
6. Report Template ................................................................................................................................ 9
How should the toolkit be used? ............................................................................................................... 9
FEASIBILITY ANALYSIS CHECKLIST ............................................................................................... 10
Introduction .............................................................................................................................................. 10
TAY PSS/FSS Evaluation Feasibility Analysis Checklist ........................................................................ 11
1. Local TAY Programs and Consumers ............................................................................................. 11
2. Data Collection, Entry, and Management ........................................................................................ 12
3. Data Analysis and Reporting ........................................................................................................... 13
4. Evaluation & Quality Improvement (QI) Experience/Capacity ......................................................... 14
Scoring .................................................................................................................................................... 15
Overall and Table-Specific Score Calculation ..................................................................................... 15
Score Interpretation ............................................................................................................................. 15
Feasibility Issues and Alternate Strategies ............................................................................................. 16
1: Local TAY Programs and Consumers Issues and Alternate Strategies .......................................... 16
2: Data Collection Issues and Alternate Strategies ............................................................................. 16
3: Data Analysis and Reporting Issues and Alternate Strategies ........................................................ 18
4: Evaluation/Quality Improvement (QI) Experience/Capacity Issues and Alternate Strategies ......... 18
TAY PSS/FSS EVALUATION GUIDE AND PLANNING TOOL .......................................................... 19
Evaluation Project Information ................................................................................................................ 19
Section 1: Introduction ............................................................................................................................. 20
1a. Problem Statement ........................................................................................................................ 20
1b. Literature Review .......................................................................................................................... 20
1c. Evaluation Goal and Aims ............................................................................................................. 22
1d. Significance ................................................................................................................................... 22
Section 2: Methods .................................................................................................................................. 24
2a. Study Design ................................................................................................................................. 24
2b. Rationale ....................................................................................................................................... 25
2c. Study Population and Sampling Technique ................................................................................... 25
2d. Measures and Indicators ............................................................................................................... 26
2e. Data Entry/Management Plan ....................................................................................................... 31
Section 3: Data Analysis ......................................................................................................................... 34
3a. Process Evaluation Data Analysis Plan (Program Manager and PSS/FSS Surveys) ................... 34
3b. Outcomes Evaluation Data Analysis Plan (MHSIP/YSS-Youth/YSS-Family and Supplementary
PSS/FSS Forms) ................................................................................................................................. 35
REPORTING THE FINDINGS .............................................................................................................. 37
Report Template ...................................................................................................................................... 37
Executive Summary ............................................................................................................................ 37
Introduction & Methods ....................................................................................................................... 37
Results ................................................................................................................................................ 37
3
Discussion: Interpretation of the Findings, Dissemination, and Next Steps ............................................ 53

Process Evaluation Findings- Discussion ........................................................................................... 53
Outcomes Evaluation Findings- Discussion ........................................................................................ 53
Linking the Process and Outcomes Evaluation Findings .................................................................... 54
Limitations ........................................................................................................................................... 54
Next Steps ........................................................................................................................................... 55
TOOLKIT SUPPORT AND FEEDBACK .............................................................................................. 56

Frequently Asked Questions (FAQ) ........................................................................................................ 56
Do you have feedback or more questions? Contact the Toolkit Authors ................................................ 56
EXTERNAL EVALUATION RESOURCES .......................................................................................... 57
Designing and Implementing Evaluations ............................................................................................... 57
Data Collector and Data Entry/Management Training Resources .......................................................... 57
Data Analysis Resources ........................................................................................................................ 58
APPENDIX ........................................................................................................................................... 59
Outcomes Evaluation Measure Domains ................................................................................................ 60
Table 1: Adult MHSIP Domain Items ................................................................................................... 60
Table 2: YSS-Youth Domain Items ..................................................................................................... 61
Table 3: YSS-Family Domain Items .................................................................................................... 62
Evaluation Measures ............................................................................................................................... 63
Program Manager Peer and Family Support Specialist Survey .......................................................... 64
Peer/Family Support Specialist Survey ............................................................................................... 71
MHSIP PSS Supplement Page ........................................................................................................... 81
YSS-Youth PSS Supplement Page ..................................................................................................... 82
YSS-Family FSS Supplement Page .................................................................................................... 83

GLOSSARY/ACRONYMS
FSS: Family Support Specialist, a person with lived experience in the child and family mental health system
of care who can provide services and advice to parents of children receiving mental health services.
MHSIP: The Mental Health Statistics Improvement Program Consumer Satisfaction Survey. The MHSIP is a
state mandated survey that is collected by California county adult mental health programs twice each year.
MS: Microsoft, referring to the Microsoft Office Suite of software products
Outcomes Evaluation: An evaluation project that seeks to measure and assess the impact of a program on
consumers wellbeing. In the case of mental health services, outcomes may include changes in symptom
severity, recovery, functioning, quality of life, social connectedness, achievement of age-appropriate
milestones (e.g., education, employment), or consumer satisfaction.
Process Evaluation: An evaluation project designed to assess how a program operates, if a program is
meeting its operational goals, and operational barriers. A process evaluation collects data on how many
consumers a program reaches and how long each consumer is exposed to the program (i.e., engagement,
duration), how many program staff are trained and work on the program, the number of program sessions
held, and operational barriers such as the supply of program staff.
PSS: Peer Support Specialist, a person with lived experience in the mental health system of care who can
provide services and advice to people receiving mental health services.
TAY: Transition Age Youth, refers to youth consumers of mental health services age 16-25 who may be
included in child or adult mental health systems of care.
UCSD: University of California, San Diego, where the TAY Evaluation project was conducted.
QI: Quality Improvement, a systematic process of collecting information and data to study and improve
programming and procedures.
YSS: Youth Satisfaction Survey. The YSS a state mandated survey that is collected by California county
child and family mental health programs twice each year. There are two versions of the YSS; the YSS-Youth
for children, adolescents, and young adults receiving services in the child and family mental health system,
and the YSS-Family for parents and guardians of children receiving services in the child and family mental
health system.
INTRODUCTION
What is the aim of this toolkit, and who should use it?
This toolkit is designed to provide a framework for California counties, agencies, organizations, and
programs to conduct a process and outcomes evaluation of services provided by Peer and Family Support
Specialists (PSS/FSS) for Transition Age Youth (TAY). A process evaluation is meant to assess how a
program operates, if a program is meeting its operational goals, and operational barriers. Process
evaluations collect data including how many consumers a program reaches and how long each consumer is
exposed to the program (i.e., engagement, duration), how many program staff are trained and work on the
program, the number of program sessions held, and operational barriers such as the supply of program staff.
An outcomes evaluation seeks to measure and assess the impact of a program on consumers wellbeing. In
the case of mental health services, outcomes may include changes in symptom severity, recovery,
functioning, quality of life, social connectedness, achievement of age-appropriate milestones (e.g.,
education, employment), or consumer satisfaction.
The process and outcomes evaluations described in this toolkit aim to:
Process Evaluation Aim: Identify characteristics, issues, needs, and gaps in the provision of services,
training, and supply chain for TAY PSS/FSS using online surveys for program managers and PSS/FSS.
Outcomes Evaluation Aim: Assess the differences in outcomes and satisfaction with services between
TAY consumers/caregivers who have interacted with a PSS/FSS compared to those who have not using
data from the Mental Health Statistics Improvement Program (MHSIP) Consumer Survey, the Youth Services
Survey (YSS-Youth and YSS-Family), and PSS/FSS-specific one page supplements added to these surveys.
When is it appropriate to use this toolkit?

This toolkit is meant to be used by mental health program administrator and managers at the county and
provider level. Organizations that use this toolkit must have youth in the TAY age range (i.e., 16-25) as a part
of their service population, and must include some services that are provided by PSS/FSS, or plan to use
PSS/FSS in the near future. For more information on when it is appropriate to use this toolkit, see the
Feasibility Analysis Checklist on pages 10-18.
Why was this toolkit developed?

This decision to develop this toolkit is the result of an assessment of TAY Mental Health Services in
California a decade after the implementation of the Mental Health Services Act (MHSA) on January 1, 2005.
The MHSA generated increased funding to expand existing Mental Health services, develop new, innovative
programs, and improve services for at-risk groups in California, one of which is TAY. The MHSA also
established a framework for increased evaluation and quality improvement activities and greater
consumer/stakeholder voice in all aspects of mental health service development and delivery. The TAY
Evaluation Project, led by faculty at the University of California, San Diego (UCSD), sought to describe the
current state of MHSA-funded TAY mental health services and programs in California, with a special
emphasis on evaluation and quality improvement. This evaluation toolkit is part of a set of technical
assistance components designed to improve and promote TAY-specific evaluation and quality improvement
in California, based on findings from the assessment. The TAY Evaluation and Quality Improvement (TEQI)
Summit, held in October of 2015 at UCSD, was another component of the technical assistance provided as
part of the TAY Evaluation Project. More information about the UCSD TAY Evaluation Project, including
findings factsheets, TEQI Summit presentations, and more is available at www.teqisummit.org.

The 2014-2016 UCSD assessment project drew on the following data sources:
-
-
-
A literature review of the evidence base for TAY-specific mental health services
Phone interviews with 39 California county Behavioral Health Services administrators
Online surveys with 180 providers who serve TAY across California
The decision to develop a TAY-specific evaluation toolkit was made based on multiple findings of the
assessment regarding a lack of TAY-specific quality improvement activities at the county and provider level,
and providers reported technical assistance needs:
Lack of TAY-specific Evaluation and Quality Improvement Activities at the County Level.
-
-
-
33% of county administrators described conducting TAY-specific process evaluations, compared to

88% of providers.
77% of county administrators described conducting TAY-specific outcomes evaluations, compared to
92% of providers.
Only 44% of county administrators described having TAY-specific quality improvement activities,
compared to 100% of TAY providers.
TAY Providers Technical Assistance Needs. When asked to identify needed technical assistance for their
TAY programs, 47% of providers indicated they wanted assistance identifying effective programs for TAY,
34% wanted assistance designing or conducting program evaluations, and 27% wanted assistance with
quality improvement projects.
The topic of this evaluation toolkit, assessing the impact of peer-support specialists on TAY consumers self-
reported outcomes and satisfaction with services, was chosen based on the following findings:
There has been a nationwide push to incorporate peer support into behavioral health services over
the last decade since the Presidents New Freedom Commission Report in 2003. Peer-support programs
were also advocated in the MHSA as part of a larger movement towards recovery-oriented and consumer-
driven Mental Health services in California. Newly proposed legislation in California, Senate Bill 614, has
been proposed to standardize training practices and core competencies, and includes language to allow
expanded billing for services performed by PSS/FSSs.
TAY peer support programs are relatively common in California; 54% of providers had peer-led
programs for TAY, including support groups (44%), education services (23%), and counseling programs
(19%). TAY Clubhouses found in many counties are a good example of a peer-led TAY service.
However, the literature review indicated that little has actually been done to measure consumer
outcomes of peer-led programs, especially for TAY consumers. Consequently, strong evaluation and
quality improvement activities are needed to ensure program effectiveness at the local level, which may
eventually lead to the development of a stronger evidence base for peer support programs.
An advisory group composed of state, county, provider, and consumer stakeholders also supported the
development, implementation, and interpretation of the assessment and subsequent findings. The advisory
group felt that a TAY-specific evaluation toolkit would be an efficient way to disseminate technical assistance
in a format that would appeal to many counties and providers, especially those that struggle to identify, plan,
and implement evaluation projects.
What does this toolkit include?

Here are the components of the toolkit in the order in which they should be used:
1. Feasibility Analysis Checklist

The Feasibility Analysis Checklist (pages 10-18) allows you to assess your organizations current resources
(e.g., personnel, skill base, capacities) and determine if and how you should proceed with the evaluation
described in this toolkit. Suggestions for alternate strategies in areas where resources may be insufficient are
provided.
2. Step-by-Step Evaluation Guide and Planning Tool
The Step-by-Step Evaluation Guide and Planning Tool (pages 19-36) is designed to support the
implementation of the evaluation project described in this toolkit, adapted to your organizations needs and
resources. This guide and planning provides instructions and options for adapting each section to your
organizations needs and resources. It also includes sample text for users to write their own evaluation plan
or protocol document.
3. External Resources
This section on external resources (pages 57-58) to help you implement your evaluation includes a list of
evaluation-related manuals and guides, and training resources such as video and web-based tutorials that
can help staff teach themselves how to handle parts of the evaluation that they have little experience in, such
as training data collectors or conducting basic statistical analysis using Microsoft Excel.
4. Evaluation Survey Measures

The evaluation described in this toolkit relies on surveys collected from TAY. Each year, the state of
California releases the MHSIP and YSS surveys to collect data on outcomes and satisfaction from
consumers in the adult and child/family systems of mental health services. Up-to-date surveys are available
on the California Institute for Behavioral Health Solutions (CIBHS) website: http://www.cibhs.org/consumer-
perception-surveys. Usually, all consumers who receive services during a specified time (e.g., one weeks)
are asked to complete the MHSIP/YSS surveys. We chose to use these surveys as the part of the evaluation
project data collection because they are state mandated; all counties must collect these surveys twice a
year. Because of this, counties and organizations should already be familiar with the process of collecting
these surveys, and can incorporate the evaluation into one of these two annual data collection periods
efficiently.
For this evaluation, you should use the versions of the MHSIP/YSS released by the state the same year you
plan to implement this evaluation. This toolkit includes copies of the following additional survey measures
that are used in the evaluation (see Appendix- Evaluation Measures, page 63):
-
-
-
-
-
Program Manager PSS/FSS Survey (Multiple pages)

PSS/FSS Survey (Multiple pages)
MHSIP Supplementary PSS Satisfaction Survey (1 page)
YSS-Youth Supplementary PSS Satisfaction Survey (1 page)
YSS-Family Supplementary FSS Satisfaction Survey (1 page)
The MHSIP/YSS Supplementary PSS/FSS Satisfaction Surveys should be added to the beginning of the
MHSIP/YSS for any TAY age 16-25 or their caregivers who take the surveys. The program manager and
PSS/FSS survey should be collected from program staff the same month as the MHSIP/YSS. More
information on the surveys can be found in the Step-by-Step Guide in this toolkit (pages 19-36)
5. Double Data Entry Form Template

This low-tech data Microsoft Excel Workbook entry form allows manual data entry of paper surveys with
built-in quality control measures. This form may be especially useful for organizations who lack data entry
and management systems, and those who have difficulty accessing data from mandatory state surveys. The
8
workbook includes an instruction tab, two tabs to enter all of the data twice, and a final tab that highlights
discrepancies between the two data entry tabs. This allows the user to verify the entries using the paper
surveys, improving the quality of the data for analysis. Once consumer data is added to this form, it should
be password protected to ensure consumer confidentiality. This template is provided as a supplementary file
included with this toolkit.
6. Report Template
The Report Template provided in this toolkit (pages 37-55) is designed to help users report and interpret the
findings. It includes table templates, which are formatted, labeled data tables with empty cells that can be
filled in with numerical data, such as means or percentages. The Report Template also includes headings
and subheadings as a suggested organizational style, and each section includes a list of questions to be
answered in that section. The discussion sections provide questions that users can answer in their report to
help interpret the findings from the process and outcomes evaluations, synthesize recommendations, and
develop next steps for your PSS/FSS programs for TAY.
How should the toolkit be used?

The Figure 1 illustrates the steps to using the TAY PSS/FSS Evaluation Toolkit.

Figure 1: Summary of Steps to Using the TAY PSS/FSS Evaluation Toolkit
1. Read the entire Toolkit Document
To familiarize yourself with the project
2. Complete the Feasibility Analysis Checklist (Pages 10-18)

To assess if and how you should conduct the evaluation
3. Use the Step-By-Step Evaluation Guide and Planning Tool (Pages 19-36)
To develop and implement your own evaluation plan
4. Use the External Resources section (Pages 57-58)

To access additional information on how to plan the evaluation, train data collectors, and enter, analyze, and
interpret the data.
5. Use the Evaluation Surveys (Appendix, pages 59-83)

To collect data for the evaluation
6. Use the Double Data Entry Form (Supplementary Document)

To enter the data from the surveys for analysis
7. Use the Report Template (Pages 37-55)

To report and interpret the findings from the evaluation
FEASIBILITY ANALYSIS CHECKLIST

Introduction
What is the feasibility analysis checklist? This checklist provides a framework to help users determine if
and how to proceed with the TAY PSS/FSS Evaluation Toolkit. If none of your TAY programs include
PSS/FSS services, this evaluation is not feasible for your organization at this time. Each table in this
checklist includes questions which represent important skills, resources, or capacities needed to implement
the project described in the TAY PSS/FSS Evaluation Toolkit in the following areas:
1.
2.
3.
4.
Local TAY Programs and Consumers

Data Collection, Entry, and Management
Data Analysis and Reporting
Evaluation/Quality Improvement (QI) Experience/Capacity

Who should complete the feasibility analysis checklist? The person who completes these feasibility
analysis checklist questions should be knowledgeable about their organizations TAY programs, staff
availability, and organizational capacity. This may be someone with quality improvement or evaluation
responsibilities or a lead program manager. The person who completes the checklists should also be the
person who intends to lead the TAY PSS/FSS Evaluation project at your organization. Other staff may need
to be consulted to answer some of the questions accurately.

How should the feasibility analysis checklist be completed? Users should answer each question to the
best of their knowledge by circling one of the answer choices, and complete the scoring instructions at the
conclusion of each checklist. An arrow () before a question in each table indicates a follow up question
from the previous question. The last column of each table is optional, and may be filled in as a reference in
planning the evaluation project and adapting the Evaluation Planning Tool, which serves as a template for
the evaluation plan.

How are these questions scored and interpreted? An additional section on final scoring instructions,
interpretation, and suggested alternate strategies begins on page 10. Using the scores from the checklists
and the overall score, users will be able to determine if and how they should proceed with the TAY PSS/FSS
Evaluation project. Scoring ranges and options for the decision to proceed include:

75%-100%: Proceed; alternate strategies are optional, but may be helpful

25%-74%: Proceed with caution, and incorporate some alternate strategies
0-24%: Use alternate strategies or do not proceed

A list of alternate strategies is provided for each checklist area, which may help you adapt the toolkit and
implement the evaluation project, based on your organizations resources. If you have a situation that is not
mentioned in the feasibility analysis checklist that you feel will affect your ability to carry out the evaluation
described in the toolkit, contact the toolkit authors to discuss this situation and possible solutions via email at
teqisummit@gmail.com or by filling out the feedback form on the TEQI Summit website at
http://www.teqisummit.org/contact.html. Someone else will likely face the same problem at some point, and
this form/toolkit can be updated to reflect these situations.
10
TAY PSS/FSS Evaluation Feasibility Analysis Checklist

1. Local TAY Programs and Consumers
Please circle your answer for each question, and write in the actual number if applicable.
Questions

Circle your answer
Write in your answer (optional)
1. How many programs do you
have that include TAY? These
can be programs exclusively for
TAY, or those that include TAY
with adult or child populations.
1-4
5+
Actual number of programs that

include TAY: _______
2. How many of these programs

are exclusively for TAY?
1-5
5+
Actual number of programs exclusively

for TAY: _______
0-49
50-99
100+
Actual number of TAY served: ______
4+
2-3
List languages:

1-4
5+

Actual number of TAY

programs/services that use PSS/FSS:
_____
3. In total, how many TAY

consumers does your
organization work with?
4. In how many languages (including
English) do you typically have to
provide TAY programs/materials?
5. How many of your TAY programs

use peer support specialists
(PSS) or family support
specialists (FSS)?
6. In total, how many TAY receive

0-30
31-99
services from PSS/FSS through
these programs?
7. For comparison purposes,
0% (none) Less than
what is your best estimate of the
or 100% 25% (a few)
percent of current TAY
(all)
or more
consumers who have interacted
than 75%
with a PSS/FSS through your
(most)
programs?
8. In total, how many PSS/FSS
0
1-20
work with TAY in these programs?
Responses
Write the number of circled responses in
each column
Scoring
Totals
Multiply the two numbers above each of
the following cells together and write the
product
100+
Actual number of TAY served by

PSS/FSS: ______
Between
25% and
75%
Percent of TAY served by PSS/FSS:

______
21+
Actual number of TAY PSS/FSS staff:

______

___
____
____
x 0
x 1
x 2

=____

=____

=____

Sub-score Total: _____/16
Add the numbers in the four cells to
the left together, and write the sum on
the line above.
11
2. Data Collection, Entry, and Management

Questions

Circle your answer
1. How many staff can help with
data collection (i.e., collecting
surveys)?
2. How much data collection
experience does your staff have?

3. How much of your staff has
received training in data
collection?
0-2
3-5
6+

None
Some
A lot

None
Some
Many

Some
A lot

Sometimes
Yes
4. To what extent do you have access

None
to supplies/ equipment to
administer paper-based
surveys? (e.g., copiers, paper,
pens/pencils)

5. Does your team regularly collect
No
the state mandated MHSIP/YSS?

6. Can your team access
No, we
datasets from the MHSIP/YSS?
never get
data
7. How long does it take your

team to get MHSIP data?
8. How many staff do you have who
can do data entry (i.e., entering
survey responses into a
spreadsheet or database)?
9. How much IT support do you have
for data management?
Responses
each column
Scoring
Totals
product
We get
some data
We get
complete
datasets or
enter the
data
ourselves
We do not More than 6 Less than 6
get the
months
months
data

0
1-2
3-4

None
Some
A lot

___
____
___
x 0

=____

x 1

=____

x 2

=____

Actual number of staff available for

data collection: __________
Briefly describe experience:

Briefly describe training:

Briefly describe:
Briefly describe:
Describe this process:
Length of time it usually takes your

team to get MHSIP/YSS datasets for
analysis, if at all:

Actual number of staff available to
enter data: __________
Briefly describe:

the left together, and write the sum on
the line above.
12
3. Data Analysis and Reporting

Questions

Circle your answer
1. How many staff do you have with
intermediate to advanced
statistical analysis skills (i.e., can
assess statistical
significance/conduct hypothesis
tests).
2. How much time would these
staff be able to dedicate to
analyzing data from this evaluation
project?
3. How familiar are these staff

with evaluation analysis and
reporting?
1-2
3-4

Actual number of staff with statistical

analysis skill: ________
None
Some
A lot

Briefly describe:
None
Some
A lot

Briefly describe their experience:
4. What kind of software to conduct None, and

Basic
Advanced Name software:
statistical analyses/reporting do do not have (e.g., can
(e.g.,
you have available?
training/ use or have
SPSS,
staff to use used Excel
SAS,
Excel for
for this)
STATA, R)
analysis

5. To what extent do you have

relationships with institutions
who can do statistical analysis, if
needed (e.g., universities,
colleges, research institutions, or
consultants)?
Responses
each column
Scoring
Totals
product
None
Some
A lot

___
____
____
x 0

=____

x 1

=____

x 2

=____
Briefly describe:

Add the numbers in the four cells to the
left together, and write the sum on the
line above.
13
4. Evaluation & Quality Improvement (QI) Experience/Capacity

Questions

Circle your answer
1. How would you describe the level
of your evaluation/QI activities to
date?
None
Basic
2. To what extent have these

evaluation/QI activities been TAY-
specific?
None
Some
A lot

3. How many staff do you have that

are involved in evaluation/QI
activities?
1-3
4+

4. To what extent do you have a

framework or model of quality
improvement in place that you use
consistently with your programs?
None
5. To what extent have you been able

to implement recommended
changes from past evaluation/QI
projects AND consequently
observe improvement?
None
Responses
each column
Scoring
Totals
product
Advanced Briefly describe:

Briefly describe
Actual number of staff involved in QI:

_______
Some, we
A lot, we Briefly describe framework or name
have a basic
have a
QI Model:
framework or specific,
use
detailed QI
elements
model
from

different
models
Some
A lot
Briefly describe or give an example:

___
____
____
x 0

=____

x 1

=____

x 2

=____

the left together, and write the sum
on the line above.
14
Scoring
Overall and Table-Specific Score Calculation
Use this table to calculate percent (%) scores by dividing each of the scores from the first six checklist tables
by the total possible points for each area. For example, if your score from the questions in the table labeled
3. Data Analysis and Reporting was 5 out of 10 possible points, your percent score would be 50%.

Scores
% Score
Write in from
Domain
previous pages Possible Points
Maximum: 100%
1. Local TAY Programs and Consumers
_____
16
_____%
2. Data Collection, Entry, and

Management
_____
18
_____%
3. Data Analysis and Reporting
_____
10
_____%
4. Quality Improvement (QI)

Experience/Capacity
_____
10
_____%
Sum of numbers in
this column:
Sum of numbers in
this column:
In each row, divide score by

possible points and multiply by
100
_____
54
_____%
Instructions
TOTALS AND OVERALL SCORE

Score Interpretation
Use the following table as a guide to determine how feasible your implementation of the TAY PSS/FSS
Evaluation Toolkit will be, based on your overall % score from the implementation checklists. For example, if
your overall % score calculated above was 60%, your project would fall in the yellow range, and you should
proceed with caution, and incorporate some alternate strategies.
% SCORE RANGE
OVERALL FEASIBILITY INTERPRETATION
75%-100%
Proceed; alternate strategies are optional, but may be helpful
25%-74%
Proceed with caution, and incorporate some alternate strategies
0%-24%
Use alternate strategies or do not proceed

If one of your scores is between two of the score range categories, or close? (e.g., 20-30%), consider the
interpretations for both % score ranges (e.g., 25-50% and 50-75%), and adopt alternative strategies as
appropriate for your situation.

The following pages provide suggested alternate strategies by each checklist domain, so that you can tailor
your project to meet your needs and organizational capacities. Look back at each checklist while examining
the issues and alternate strategies, and determine which of the issues most closely reflects your own
situation in designing and implementing the TAY PSS Evaluation described in the toolkit.
15
Feasibility Issues and Alternate Strategies

1: Local TAY Programs and Consumers Issues and Alternate Strategies
Not enough TAY or TAY-specific programs to include? If you have fewer than 50 total TAY consumers in
your programs, consider teaming up with another nearby county or organization to increase your sample
size, and conduct the evaluation jointly. Remember to include results and recommendations specific to your
own county or organization in your report.

Few or no TAY-exclusive programs? If you only have TAY that are involved in programs with adult and
child populations, and no programs just for TAY, consider collecting and analyzing data from just these TAY,
or TAY and adults for comparative purposes. Ensure that in your data collection, you can distinguish which
participants are TAY and which are adults by age for analysis purposes, and include this comparison in your
report.

Too many languages in use for TAY programming? Translating data collection instruments can take up
valuable time and staff effort, so consider limiting your evaluation to one or two language groups (e.g.,
English and Spanish). Acknowledge this limitation in your reporting, and consider future projects with other
language groups in the coming years.

Few programs that provide TAY with PSS/FSS services or few TAY receiving PSS/FSS services?
If you have fewer than 30 TAY consumers in your programs who receive services from PSS/FSS, you may
have difficulty getting useful results from the evaluation. Consider teaming up with another nearby county or
provider to increase your sample size, and conduct the evaluation jointly. Remember to include results and
recommendations specific to your own county or organization in your report.

Percent of TAY who receive PSS/FSS services vs. those who do not inadequate for comparison
purposes? The design of the evaluation proposed in the toolkit requires only one data collection (i.e., cross-
sectional, instead of pre-post), but also requires that you have some TAY who have interacted with a
PSS/FSS in the past six months, and some who have not. If you have fewer than 100 TAY consumers total
in your programs, and all/most (over 75%) of your TAY currently interact with PSS/FSS, or all most do not
interact with PSS/FSS (less than 25%), this evaluation design may not work for you because there are not
enough TAY in either group to be able to compare them. Consider turning the evaluation into a pre-post
evaluation, where you administer surveys at intake and then again six months later (or at the mid point and
end of a program) and look for changes in outcomes and satisfaction over time.

2: Data Collection Issues and Alternate Strategies
Few staff trained/available to conduct data collection, or staff are already overburdened with
assessments? The evaluation described in this toolkit uses a single page attached to the state-mandated
MHSIP/YSS surveys, with the assumption that counties and providers have already allocated staff and
resources to collect those surveys, and are familiar with the survey process. If you still lack staff to implement
the surveys with consumers, consider recruiting and training youth volunteers or local college students to
collect surveys. The training needed is not extensive, and youth tend to have greater flexibility in terms of
scheduling to accommodate data collection projects. Volunteer groups that your organizations have worked
with in the past or who have worked with mental health issues and college undergraduates majoring in social
sciences, such as psychology, sociology, economics, or social work, may be especially well suited. Follow
your organizations procedures for handling volunteers or student workers. You may also wish to contact
administrative staff or faculty in relevant college departments for help recruiting and coordinating volunteers
or student workers. Links to training tutorials and presentations are provided in the Evaluation Toolkit
document in the section on External Evaluation Resources (pages 57-58).
16

Previous attempts to collect data from TAY have been met with resistance from TAY or providers?
TAY can be a challenging group to collect data from due to multiple factors. Providers may also be hesitant
to implement another assessment for fear of alienating or boring their consumers. To address this, we have
made the supplementary MHSIP/YSS survey pages as short as possible. Each group of TAY consumers
may vary in their opinions about completing assessments. To address this on a local level, you could also
hold an informal focus group with TAY and/or providers to discuss how to make data collection easier for
everyone, such as identifying convenient times, locations, and potential incentives for participation.

Little/no access to paper-based survey resources? Counties with fewer resources may be challenged in
having access to a reliable printer or copier, may have limited funds to purchase supplies, or may even lack
the physical space and furniture to comfortably administer surveys to TAY. One option is to consider limiting
the number of TAY you collect data from to reduce the use of these resources. However, serious concerns
about insufficient resources to conduct paper-based assessments represent a greater need beyond the
scope of the evaluation toolkit.

Difficulty accessing MHSIP/YSS data in a timely fashion? If you do not have access to MHSIP/YSS
datasets after sending it to the state, or these datasets take a long time to receive, consider talking to
MHSIP/YSS state administrators about ways to facilitate access to the data. Alternately, you can explore
increasing your countys or organizations capacity to enter data, and scan the MHSIP/YSS forms before
sending them to the state. County staff can then enter the forms into a database manually using the MS
Excel template included in the toolkit (Double Data Entry Form), or invest in software such as TeleForm,
which is what the state uses to enter MHSIP/YSS data. See the TAY PSS/FSS Evaluation Toolkits Section
2e on Data Entry for more information on alternative ways to enter and access data.

No/few staff trained in or available for data entry? Data entry requires little training and is a relatively
tedious, repetitive task. You may be able to recruit volunteers or local college student interns to enter data.
Volunteer groups that your organizations have worked with in the past or who have worked with mental
health issues and college undergraduates majoring in social sciences, such as psychology, sociology,
economics, or social work, may be especially well suited. Follow your organizations procedures for handling
volunteers or student workers. You may also wish to contact administrative staff or faculty in relevant college
departments for help recruiting and coordinating volunteers or student workers.

Limited access to data entry/management software or no data entry quality checks in place?
Some counties and providers have access to sophisticated data entry software to turn survey responses into
databases that can then be analyzed, and some do not. Another challenge is ensuring that the data from
paper-based assessments is entered accurately; it is common for even experienced data entry professionals
to make mistakes sometimes. For those who lack data entry and management systems or data entry quality
checks, the TAY Evaluation Toolkit includes a MS Excel Double Data Entry workbook template. This
template includes instructions on entering the data twice into separate worksheets, and reconciling
discrepancies between the two datasets to ensure the data is entered as accurately as possible.

Limited IT support available for data management? You may have access to data entry and management
software, but lack IT capacity or support to make modifications or create new programs to enter data from
the evaluation assessments. Or, if you have access to this support, it may take an extended period of time to
implement necessary changes or receive other support as needed.
If this is the case, you may consider using the MS Excel Double Data Entry Form, described in the previous
paragraph. The template is relatively simple, easily modified, and requires only basic knowledge of Excel.

17
3: Data Analysis and Reporting Issues and Alternate Strategies

No/few available staff with statistical analysis skills or who are able to assess statistical
significance/conduct hypothesis testing? The Evaluation Toolkits Section 3. Data Analysis and
Reporting (page 34) includes alternate analysis plans at different levels of statistical ability to accommodate
counties and providers with limited capacity in this respect. You can also consider seeking Masters level
student interns from fields such as Public Health, Psychology, Social Work, Sociology,
Mathematics/Statistics, or other relevant disciplines who can conduct appropriate statistical analyses for your
evaluation as an independent study. To do this, use your local professional network (e.g., any local university
faculty or staff you know) or contact administrators at relevant departments at local colleges and universities.
At minimum, candidates should be well-versed in intermediate level statistical analyses, including hypothesis
testing and assessing statistical significance through t-tests, analysis of variance (ANOVA), chi-square, or
other similar statistical tests. Preferably, they should have some experience in mental health research or
program evaluation. Ask candidates to provide a sample of their recent work to help you assess if they will
be able to conduct the analyses. Another option is to use websites for simple data analysis calculations.
Initiate an internet search (e.g., Google) using terms such as ANOVA calculator, t-test calculator, or chi
square calculator and use descriptive statistics such as counts, means, frequencies, and standard
deviations to assess significant differences between groups.

Software for analysis limited to Excel, or have analytic software available, but staff are not able to
use it effectively? Conveniently, it is possible to conduct hypothesis testing/assess statistical significance
using Excel. The toolkit includes resources for learning how to conduct basic but appropriate hypothesis
tests to determine if your data show significant changes in consumer outcomes over time. These resources
are complementary to the Excel Data Entry template, which will give you a useable, accurate database to
work with in your analysis.

4: Evaluation/Quality Improvement (QI) Experience/Capacity Issues and Alternate Strategies
Little or no experience doing TAY-specific evaluation/quality improvement activities? Based on
findings from the University of California, San Diego (UCSD) TAY Evaluation Study, TAY-specific evaluation
and QI activities are more likely to occur at the provider level than the county level. Counties who wish to
implement this toolkit may wish to hold meetings between county staff and provider staff to share information
about conducting evaluation/QI activities specifically for TAY. These meetings may be focused on identifying
strategies for engaging TAY in the Evaluation Planning Tool adaptation process (e.g., holding a meeting with
TAY peer leaders to explain the evaluation to them and get their feedback), how to successfully administer
surveys to TAY consumers, and successes and challenges of past TAY program evaluation/QI projects.

Past attempts at evaluation or quality improvement (general or TAY-specific) have been limited or
have not been useful? To an extent, the guidance provided in these toolkits is designed to increase
counties and providers capacity to conduct comprehensive TAY-specific evaluation and QI activities, from
conception to conclusion. Reviewing feedback and lessons learned on any past evaluation projects is an
important step in identifying and developing strategies to address any issues in your evaluation plan.

18
TAY PSS/FSS EVALUATION GUIDE AND PLANNING TOOL

This portion of the toolkit includes a description and instructions for each section of the evaluation, and
sample text for users to develop their own evaluation plan or protocol. Directions for writing each section are
included so that users can tailor their evaluation plan to your organizations characteristics and context.
Options have been included in some of the sections sample text to allow for flexibility in the level of
methodological rigor and skills and/or resources needed to accommodate organizations with greater or fewer
resources in these areas.
Evaluation Project Information

Description and Instructions
This section describes your evaluation and evaluation team, including the Mental Health Plan (MHP) name,
the title of the project, the project leader and his or her title and role, the initiation dates and completion
dates of the evaluation. The table in the sample text for this section includes the names, titles, affiliations,
and roles of the evaluation team. The evaluation team should include individuals with diverse skills,
knowledge, and perspectives who can support the work that goes into a successful evaluation. In addition to
organizational staff with experience in evaluation or quality improvement, the team should include subject
matter experts, such as staff from provider organizations that offer PSS/FSS services, PSS/FSS
themselves, and TAY or their caregivers who use PSS/FSS services and those who do not. The example
provided in the sample text can serve as a guideline of what roles the members of a multi-functional team
may hold. You should consult the evaluation team multiple times throughout the evaluation for their input on
the design, implementation, and interpretation of the results.

Sample Text
Evaluation Project Information

Organization: [Organization Name]

Evaluation Project Title: TAY Peer and Family Support Services: Process and Outcomes Evaluation

Start Date: MM/DD/YYYY
End Date: MM/DD/YYYY

Evaluation Team Members
First and Last
Name, Degrees (if
applicable)
Title/Role
Jane Doe, MSW
Principal administrative
analyst

Peer Support Specialist

Peer Support Specialist

TAY Consumer or Family
Member

TAY Consumer or Family
Member
Role in Evaluation
Evaluation Manager
Evaluation Assistant
Data Entry
Statistician
Consultant
PSS Advisory Group Member
PSS Advisory Group Member
TAY Advisory Group Member
TAY Advisory Group Member
Affiliation
[Organization
Name]

Phone/Email
XXX-XXX-XXXX
jane.doe@orgname.org

19
Section 1: Introduction
1a. Problem Statement
The first paragraph in this section should state the problem to be addressed by this evaluation. It is important
to understand how your organizations TAY PSS/FSS services function, and how delivering services through
PSS/FSS affects TAY outcomes and satisfaction. This toolkit outlines an evaluation that will allow users to
address this problem.

Sample Text
Despite movements at the national and state level in California to incorporate Peer and Family Support
Specialists (PSS/FSS) into Behavioral Health Services, there is a remarkable lack of evidence regarding
outcomes for consumers who receive PSS/FSS services, especially for TAY consumers and their caregivers.
Peer/Family support is one of the new service areas promoted for TAY as part of the 2004 Mental Health
1
Services Act (MHSA) to improve TAY engagement in services and address TAY as an underserved population.
Consequently, peer/family support now plays a major role in behavioral health services for TAY in California,
with over half (54%) of MHSA-funded TAY programs incorporating some form of peer-led support groups,
2
education, and/or counseling.

1b. Literature Review
The paragraphs that follow the problem statement should include a literature review, which conveys
information about:
(1) Recent national and state-level policies and trends advocating for increased use of PSS/FSS as an
evidence-based practice, especially for TAY.
(2) How published literature reviews on consumer outcomes of PSS/FSS services highlight a lack of
rigorous evidence, and that the only finding with a degree of consistency across multiple studies is
that consumer outcomes for most PSS/FSS-led services are not significantly different (i.e., no better
or no worse) than consumer outcomes for the same services provided by clinicians
(3) A brief review of more recent research available on PSS/FSS consumer outcomes (2012-2015),
noting continued mixed findings and lack of methodological rigor.
(4) How the literature on PSS/FSS is even more scarce for TAY, and why evaluation is needed at the
local level to ensure that TAY benefit from these services.
Read the sample text closely, and add any additional relevant information to your evaluation plan as you see
fit. Citations are included after the sample text. You may wish to consult the recent original articles cited in
the sample text to familiarize yourself with the evidence base for peer services, especially the review articles
3,4
by Chinman et al., 2014 and Lloyd Evans et al., 2014.

Sample text
In the 12 years since the Presidents New Freedom Commission Report in 2003, there has been a nationwide
push to incorporate peer support services into many levels of behavioral health services. Published literature,
nationwide evidence from peer organizations, and recent policies and regulations strongly indicate that the role
of peer support services in behavioral health settings is actively growing and developing across the nation. In
2007, peer support services for adults became recognized by the Centers for Medicare and Medicaid Services
1
Jones, N., McMenamin, S., Ojeda, V.D., Gilmer, T.P. (2015) Supporting Functional Recovery among Transition Age Youth: A review of
the literature. (In progress).
2
Ojeda, V.D., Jones, N., McMenamin, S., Hurst, S., Bugdorf, J., Gilmer, T.P. (2015) Implementation of Age Specific Services for
Transitional Age Youth in California. (In-progress).
3
Lloyd-Evans, B., Mayo-Wilson, E., Harrison, B., Istead, H., Brown, E., Pilling, S., et al. (2014). A systematic review and meta-analysis
of 20 randomized controlled trials of peer support for people with severe mental illness. BMC Psychiatry, 14(1), 112.
http://doi.org/10.1186/1471-244X-14-39
4
Chinman, M., George, P., Dougherty, R. H., Daniels, A. S., Ghose, S. S., Swift, A., & Delphin-Rittmon, M. E. (2014). Peer Support
Services for Individuals With Serious Mental Illnesses: Assessing the Evidence. Psychiatric Services, 65(4), 429441.
http://doi.org/10.1176/appi.ps.201300244
20
as an evidence-based practice, and became designated as a billable and reimbursable service. However, this
evidence-based label may be due largely to the perceived cost-effectiveness, consumer satisfaction, and ease
of adding peer-led elements to existing support services. Peer support has also been advocated in California
during this time through the 2004 Mental Health Services Act (MHSA), as part of a larger goal of moving
towards recovery-oriented and consumer-driven Mental Health services. Furthermore, recently proposed state
legislation supports expanding the role and numbers of PSS/FSSs. California Senate Bill (SB) 614 has been
proposed to standardize training practices and core competencies, and includes language to allow expanded
billing for services performed by PSS/FSSs.

Despite this systemic push towards increasing the role and numbers of PSS/FSSs continues to gain
momentum, there is little literature detailing the impact of interactions with PSS/FSSs on consumers
themselves. Past literature reviews have focused on the benefits of peer support services to peer support
specialists themselves, or on engagement and utilization; PSS/FSSs have been found to be more effective than
5
traditional providers reaching demographically unserved/underserved. However, effectiveness in engagement
is distinct from effectiveness in improving consumer outcomes. Confusion over the evidence base for consumer
outcomes of peer support is best exemplified by two review articles published in 2014; the first indicated that the
evidence base was moderate, suggesting that peer support may reduce hospitalizations and lead to
improvements in multiple outcomes related to recovery, but the most salient finding was that peers were at least
6
as effective in delivering traditional services as non-peers. The second review soundly rejected the evidence
base for peer support, citing the serious methodological limitations of existing studies, and indicating that the
evidence does not warrant the policies that are being enacted globally to incorporated peer support into mental
7
health services. Therefore, before committing additional resources in this area, the lack of rigorous research
necessitates local-level evaluation and quality improvement activities to ensure that peer-support programs
have tangible outcomes among consumers.

Scant recent literature on PSS/FSS suggests mixed findings and continues to lack methodological rigor; studies
have observed improved consumer access and utilization of services in housing programs with a peer-support
8
component and that peer support may mediate the physical declines associated with depression in older adults
9
in Hong Kong. A study set in China showed improved functioning and lower re-hospitalization rates for
schizophrenia patients of families in mutual support groups compared to those in psycho-education programs or
10
usual care. Another study reported improved sense of empowerment, confidence, functioning, and alcohol
11
abuse issues among U.S. veterans in a peer-support program. However, a subsequent study replicating the
latter program found no significant differences in outcomes between consumers receiving peer support services
12
compared to those in clinician-led support groups or usual treatment. Other studies have also shown that
13,14
outcomes for consumers with PSSs were no better (and also no worse) than usual care.
In addition to these
mixed findings, all of these studies have significant design flaws, such as insufficient sample size, lack of an
appropriate comparison group, and limited generalizability.

For TAY, the literature on PSS/FSS is even more scarce, with no clear evidence on how the impact of PSS/FSS
services may differ between TAY and other age groups, such as adults, or how PSS/FSS services might be
15
optimized for TAY. Therefore, before committing additional resources in this area, the lack of rigorous
5
Repper, J., & Carter, T. (2011). A review of the literature on peer support in mental health services. Journal of Mental Health, 20(4),
392-411.
6
Chinman et al., 2014.
7
Lloyd-Evans et al., 2014.
8
Bean, K. F., Shafer, M. S., & Glennon, M. (2013). The impact of housing first and peer support on people who are medically vulnerable
and homeless. Psychiatric Rehabilitation Journal, 36(1), 4850.
9
Yeung, D. Y., Kwok, S. Y. C., & Chung, A. (2013). Institutional peer support mediates the impact of physical declines on depressive
symptoms of nursing home residents. Journal of Advanced Nursing, 69(4), 875885. http://doi.org/10.1111/j.1365-2648.2012.06076.x
10
Chien, W. T., & Chan, S. (2014). One-year follow-up of a multiple-family-group intervention for Chinese families of patients with
schizophrenia. Psychiatric Services.
11
Resnick SG, Rosenheck RA: Integrating peer-provided services: a quasi-experimental study of recovery orientation, confidence, and
empowerment. Psychiatric Services 59: 13071314, 2008
12
Eisen, S. V., Schultz, M. R., & Mueller, L. N. (2014). Outcome of a randomized study of a mental health peer education and support
group in the VA. Psychiatric Services, 63(12), 12431246.
13
Wrobleski, T., Walker, G., Jarus-Hakak, A., & Suto, M. J. (2015). Peer support as a catalyst for recovery: A mixed-methods study: Le
soutien par les pairs comme catalyseur du retablissement: une etude a methodologie mixte. Canadian Journal of Occupational Therapy,
82(1), 6473. http://doi.org/10.1177/0008417414551784
14
Simpson, A., Flood, C., Rowe, J., Quigley, J., Henry, S., Hall, C., et al. (2014). Results of a pilot 21randomized controlled trial to
measure the clinical and cost effectiveness of peer support in increasing hope and quality of life in mental health patients discharged
from hospital in the UK. BMC Psychiatry, 14(1), 114. http://doi.org/10.1186/1471-244X-14-30
15
Jones et al., 2015.
21
research necessitates local-level evaluation to understand how these peer support programs are managed and
ensure that they have tangible outcomes among TAY consumers.
1c. Evaluation Goal and Aims

The evaluation aims developed for this evaluation are brief, specific, and concisely stated. The first aim, for
the process evaluation, relies on data from program managers and PSS/FSS surveys. The outcomes
evaluation aims focus on identifying PSS/FSS consumer outcomes and satisfaction, derived from data
collected through the MHSIP/YSS-Youth/YSS-Family and supplementary PSS one-page surveys. If you
change the aims based on stakeholder/evaluation team feedback, please ensure that the changes you make
are represented consistently throughout the evaluation plan.
Sample Text
The overarching goal of this evaluation is to identify how TAY PSS/FSS services function and assess the
consumer outcomes of and satisfaction with PSS/FSS services for [organization name]s TAY consumer
population (age 16-25). This will be accomplished through the identification of PSS/FSS service, training, and
supply characteristics and issues for TAY PSS/FSSs from the perspective of program managers and
PSS/FSSs, and a comparative assessment of consumer outcomes and satisfaction among TAY who interact
with a PSS/FSS and those who do not.

Process Evaluation (PE) Aims:
- PE1. Identify characteristics, issues, needs, and gaps in the provision of services, training, and supply
chain for TAY peer support specialists.

Outcomes Evaluation (OE) Aims:
- OE1. Assess the differences in consumer outcomes and satisfaction for TAY who interacted with a
PSS/FSS compared to those who did not in the past six months.
- OE2. Assess any differences in consumer outcomes and satisfaction for TAY with different demographics
and system backgrounds who did or did not interact with a PSS/FSSs.

1d. Significance
In California, a large proportion of consumers are exposed to PSS/FSS in some way during their treatment,
and for most organizations, this proportion is likely to grow in coming years as PSS/FSS become more
widespread, which makes conducting evaluations like this important to ensure these programs are effective
and meeting their programmatic goals.

In this section, if you have the data, include information on approximately how many consumers in your
organization have interacted with a PSS/FSS, or how many of your providers/programs include PSS/FSS
services and the number of consumers served by those providers and programs (i.e. the number consumers
potentially exposed to PSS/FSS). TAY-specific data is best, but data on all consumers is acceptable too.

If you lack data, discuss any upcoming or recent policies or strategies regarding growth of PSS/FSS services
in your organization. Add additional information as you see fit on the potential impact of using the information
about PSS/FSS you will gain as a result of this evaluation to adapt policies and/or programs and services,
and how this may impact the mental health, functional status, or satisfaction of consumers served.

Sample Text
There is an underlying assumption that interaction with PSS/FSSs benefits consumers through improved
outcomes, and that TAY may benefit more than other groups due to: (1) their status as an underserved, hard-
to-engage population and (2) the purported effectiveness of PSS/FSS in improving engagement and
satisfaction. Although there is some literature to support this, specific knowledge of the impact of interaction
22
with PSS/FSSs on consumers in [organization name] will be used to verify that interaction with PSS/FSSs
enhances outcomes for consumers, identify differential impact across consumer subpopulations (e.g., TAY vs.
adults age 25 and older), and identify gaps in training and supply of PSS/FSS. Understanding these differential
impacts of PSS/FSS on consumer outcomes can help [organization name] identify meaningful strategies to
improve operational efficiency and address challenges to successful implementation of a peer support system,
including but not limited to: lack of clarity about peer support providers duties, staff resistance and exclusion
from treatment team meetings, lack of supervision and support, lack of a viable career path, and general
negative perceptions among Support Specialists (Peer Specialist Toolkit, Chinman, Henze, and Sweeney,
2013). Identifying why TAY and adults differ in how they are impacted by PSS/FSS and implementing needed
trainings, promoting certification processes, and appropriately supporting PSS/FSSs can in turn lead to better
consumer outcomes as PSS/FSS services become better defined, focused, and skill-based.

Recent data collected by [organization name] in [year] suggests that __% of youth consumers interact with
PSS/FSS. Given the systemic push towards increasing the role in numbers of PSS/FSS in the mental health
system at the national, state, and county level, this proportion is only expected to increase. Therefore,
quantifying the impact of PSS/FSS as well as differential impacts by age and other demographic
characteristics, duration of care, and usage is of prime importance to operational efficiency as the role of
PSS/FSS continues to grow. For example, if interaction with a PSS/FSS is shown to be more effective with
TAY in certain settings, situations, or levels of care, staffing allocations can be properly planned. If there are
barriers or issues related to the impact of TAY interaction with a PSS/FSS, such as a lack of age-appropriate or
adequately trained personnel, cultural competency, or ethical boundaries, these need to be identified and
addressed. If the impact of interaction with a PSS/FSS is being hindered through retention problems due to
lack of career ladders or training, these issues need to be identified as well.
23
Section 2: Methods
2a. Study Design
This section describes the study design, linking the data collection methods to each study question this
evaluation seeks to answer. The study design is cross sectional, meaning that data is only collected one time
instead of multiple times, such as before and after a program takes place. The process evaluation design is
an observational cross-sectional study that seeks to describe PSS/FSS services at your organization.

The outcomes evaluation includes a comparative design, in which consumer-reported outcomes and
satisfaction of TAY who have interacted with a PSS/FSS in the last six months will be compared to those of
TAY who have not had such an interaction. In statistics terms, the outcomes and satisfaction measures
represent dependent variables, or the primary variables of interest in the study. The independent variable,
or the variable that we want to use to understand the differences in outcomes and satisfaction, is whether or
not TAY interacted with a PSS/FSS in the past six months. You may also think of this as a stratifying
variable that will be used to separate the outcomes/satisfaction data into two groups for comparison
purposes. The hypothesis or prediction of the outcomes evaluation is that TAY who have interacted with a
PSS/FSS in the past six months will have significantly better consumer-reported outcomes and satisfaction
than those who have not.

Sample Text
To address the Process Evaluation Aim (PE1), identify characteristics, issues, needs, and gaps in the training
and supply chain for TAY peer support specialists, we will analyze data from online surveys administered to
managers of program that employ TAY PSS/FSS, and online completed by TAY PSS/FSS themselves.
To address the Outcomes Evaluation Aim (OE1) assess the differences in consumer outcomes and satisfaction
for TAY or their caregivers who interacted with a PSS/FSS compared to those who did not in the past six
months, our comparative study design will assess differences in MHSIP/YSS outcomes and satisfaction data
between TAY consumers/caregivers who interacted with a PSS/FSS with those who did not. As it is not feasible
or ethical to randomly assign consumers to interaction with a PSS/FSS, our data collection measures will ask
TAY consumers/caregivers to report if they had any interaction with a PSS/FSS in the course of receiving
services. This data point will be used to stratify outcome and satisfaction data, allowing us to determine if there
are any significant differences in outcomes and satisfaction between these two groups. An additional set of
questions for TAY consumers/caregivers who had an interaction with a PSS/FSS will deepen our understanding
of satisfaction and outcomes of these services, and facilitate the interpretation of the findings.

To address Outcomes Evaluation Aim (OE2), assess any differences in consumer outcomes and satisfaction
for TAY with different demographics and system backgrounds who did or did not interact with a PSS/FSS (i.e.,
to determine if the effect of interaction with PSS/FSSs differs by consumer characteristics), the outcomes and
satisfaction data for the study groups may be further stratified by consumer demographics, duration of care,
time in system, and primary diagnosis.
Data collected to address PE1 will be used to further contextualize and understand the findings from OE1 and
OE2, and facilitate the development of relevant and meaningful recommendations for TAY PSS/FSS services.
24
2b. Rationale
Read through this section carefully to ensure that you understand why the evaluation uses the proposed
methodology. The text added to this section provides a brief description, rationale, and justification for the
methods used in this evaluation to address the problem (i.e., a lack of data on consumer outcomes for
PSS/FSS services).

Sample Text
Process Evaluation: Online surveys are an efficient way to collect information from program managers and
PSS/FSS. These surveys can be completed at the recipients convenience, reducing the burden of data
collection on the evaluation team and the program managers/PSS/FSS.

Outcomes Evaluation: As it is not feasible to randomly assign consumers to receive services from a PSS/FSS
or have no interaction, a comparative cross-sectional study design is proposed to assess TAY outcomes of and
satisfaction with PSS/FSS interactions. To create study groups, TAY consumers or their caregivers will self-
report any contact with a PSS/FSS by means of a supplemental page added to the ongoing MHSIP/YSS-
Youth/YSS-Family consumer satisfaction survey administered twice each year. We can then compare TAY
consumers/caregivers who have interacted with PSS/FSS with those who have not in the course of their
treatment for a number of factors including satisfaction, outcomes, and other measures available in the clinical
record. Such naturally occurring field experiments are commonly used when it is not feasible to use random
assignment. It is acknowledged that the sample of TAY consumers who interact with PSS/FSS may be different
in many ways than those who do not (e.g., differing levels of care or demographics). To address this, we will
examine stratified consumer outcomes for each of these groups of interest.
2c. Study Population and Sampling Technique

The evaluation plan should define the population who will directly participate in the evaluation project,
including consumers, program managers, and PSS/FSS. Adding a data table is not required, but if you have
data available, you can use the table provided as a sample text to display your organizations data for your
TAY consumer population, stratified by system of care if that information is available or applicable. Use the
most recent data available. If you do not have access to some of these data points, remove them from the
table or replace them with other data to which you do have access, such as information about your general
consumer population. Prior MHSIP/YSS findings may be included here as well. If your access to data is very
limited, be sure to state this in this section.

This section should also describe how participants will be identified and selected to participate, which is also
known as a sampling technique. The recommended sampling techniques for the evaluation are relatively
simple. For the process evaluation, sampling is not necessary as the entire group of TAY program managers
and PSS/FSS within a county or organization will be given the opportunity to complete the survey. A survey
that collects data from all members of a target population is also known as a census survey approach,
which differs from a sample survey approach, where a subset of the target population is recruited. For the
outcomes evaluation, because the supplemental PSS/FSS surveys are attached to the MHSIP/YSS surveys
during one of the two mandated annual data collection periods, the sampling procedure is the same as for
the MHSIP/YSS surveys. All TAY consumers and/or their caregivers who receive services during the survey
period will be asked to complete the MHSIP or YSS survey with the supplementary PSS/FSS form attached.
This type of sampling is known as a convenience sample and is adequate for evaluation purposes.

Sample Text
The study population for the process evaluation will include all program managers and PSS/FSS staff of
programs that serve TAY within our [county/organization]. The program managers will answer their survey first,
and if their program employs PSS/FSS, will distribute the PSS/FSS survey to those individuals. Sampling is not
necessary, as the entire group of TAY program managers and PSS/FSS within [county/organization] will be
given the opportunity to complete the survey.
25

The study population for the outcomes evaluation will include all [organization name] TAY mental health
services consumers ages 16-25 who complete the [Month, Year] Mental Health Statistics Improvement Program
(MHSIP) Consumer Survey or the Youth Services Survey (YSS-Youth and YSS-Family) during [dates of
MHSIP/YSS administration]. These surveys will be sent to all County mental health programs with the exception
of inpatient and emergency services, and administered to all consumers who seek services during that five-day
period. This reflects a convenience sampling strategy. TAY consumers/caregivers will receive a supplementary
page as part of the MHSIP/YSS, in which they will be asked to report if they had any interaction with a PSS/FSS
during their services in the past six months.

As seen in the table below, [Organization/County Name] employs [number] program managers who work on
TAY programming, and [number] PSS/FSS who work with TAY. [Organization/County Name] serves [number of
TAY], who make up [percent] of our total consumer population. TAY make up [percent] of consumers in our
adult system of care, and [percent] of consumers in our child and family system of care. Furthermore, we
estimate that [percent] of TAY in the adult system of care and [percent] in the child and family system of care
interact with PSS/FSS.

TAY PSS/FSS Evaluation Study Population Data, [Organization/County Name]

Number of TAY program managers and proportion of among
all program managers
Adult System of Care
Child and Family System of Care

Number of TAY PSS/FSS and proportion of among all
PSS/FSS

Number of TAY consumers and proportion of TAY among all
consumers
Number/Proportion of TAY who receive PSS/FSS services
TAY Program Managers
All Program Managers
[Number (%)]
Number
[Number (%)]
[Number (%)]
Number
Number
TAY PSS/FSS
All PSS/FSS
[Number (%)]
Number
[Number (%)]
[Number (%)]
Number
Number
TAY Consumers
All Consumers
[Number (%)]
Number
[Number (%)]
[Number (%)]
Number
Number
[Number (%)]
Number
[Number (%)]
Number
2d. Measures and Indicators

In this section, the data to be collected through this evaluation should be described in detail, including the
source of the data, information on validity and reliability from previous studies using these measures, and
how the data will be collected. The MHSIP, YSS-Youth, and YSS-Family surveys can be accessed at this
website: http://www.cibhs.org/consumer-perception-surveys. We chose to use the mandated MHSIP/YSS
surveys as the foundation of the outcomes evaluation project for several reasons:
-
-
-
All California counties and many providers should be familiar with these forms and the procedures
used to collect these surveys from consumers.
The MHSIP/YSS are mandated to be collected twice each year. Adding one page to the MHSIP is
more efficient than having TAY complete an additional measure at a separate time.
Collection procedures are generally consistent across counties due to standard guidelines. This may
be helpful if toolkit users are able to meet to discuss and compare their findings.
However, we also have to acknowledge a few limitations of the MHSIP/YSS surveys. TAY may have more
difficulty completing the MHSIP or the YSS-Youth in its entirety compared to older adults due to maturity
levels or developmental issues, or may be more likely to be uncomfortable with some of the terminology
used in the surveys about mental illness. Per feedback from stakeholders, we have revised the methods to
26
place the Supplementary PSS/FSS forms at the beginning of the MHSIP survey instead of the end, to help
ensure that better quality data on PSS/FSS is collected. Toolkit users may adapt the form to include it at any
point in the MHSIP or YSS surveys, per their discretion and feelings about their TAY consumers
preferences. Furthermore, while the YSS-Family and the YSS-Youth are very similar to each other, they are
less similar to the MHSIP, which makes comparisons across older (age 18+) and younger (17 and younger)
TAY challenging. However, until more TAY-specific, easy-to-use, relatively short outcome measures are
developed, using the MHSIP/YSS for this evaluation project is reasonable. This section also includes
descriptions of the indicators (i.e., specific survey items that measure outcomes measured in our data
collection to answer the primary study questions and the secondary study questions.
For the Process Evaluation Aim (PE1), online surveys can be developed and distributed to all program
managers within the organizations participating in the evaluation. The program manager surveys assess
indicators related to PSS/FSS, including how many are employed, wages, their roles, training/certifications,
perceptions of cost effectiveness, hiring experiences/plan, consumer eligibility for PSS services, and
successes/limitations/challenges of PSS/FSS. The PSS and FSS surveys can also be developed and
distributed online, and assess indicators including the types of peer/family support services provided, the
frequency of each type of service provision, perceptions/advantages/disadvantages of working as a PSS,
trainings completed, wages and hours worked, interest in career advancement within BHS, suggestions for
improvement, gaps in services, personal impact of working as a PSS, and relationships with non PSS/FSS
staff. You can adapt these surveys as needed, but adaptations will make your findings less comparable to
other organizations who complete this evaluation.
For the Outcomes Evaluation Aims (OE 1 & 2), indicators come from the MHSIP, YSS-Youth, YSS-Family,
and their corresponding supplementary PSS/FSS one-page surveys. These include: satisfaction, access to
services, quality and appropriateness of services, involvement in treatment planning, general outcomes from
services, functioning, social connectedness, contact with law enforcement, and satisfaction/outcomes of
PSS/FSS services. These measures are included as attachments at the end of the toolkit. Please note that
the state may change these surveys slightly each time they are implemented (i.e., every six months). The
information in the sample text is based on the November 2015 MHSIP/YSS surveys. See Tables 1, 2, and 3
in the Appendix for more detail on the specific questions from the MHSIP, YSS-Youth, and YSS-Family that
will be used in the evaluation. Update the information on the MHSIP/YSS measures based on the version
you are using for your evaluation and any new information that has become available about these measures,
and add any additional measures you wish to include.
Options for Measures and Indicators
At minimum, your evaluation must include the following measures:
- MHSIP and the MHSIP PSS Supplement for TAY in the Adult System of Care
- YSS-Youth and the YSS-Youth PSS Supplement for TAY in the Child, Youth, and Family System
of Care
- Program Manager Survey for program managers of organizations that provide PSS/FSS services
to TAY and/or their families.
- PSS/FSS Survey for PSS/FSS of organizations that provide PSS/FSS services to TAY and/or their
families.
VERY IMPORTANT NOTE: The MHSIP/YSS and the Supplementary form MUST INCLUDE a participant
ID number that allows the data to be linked (or merged) together for analysis. This is because to be able
to compare TAY who have seen a PSS/FSS to those who have not, a participants answer to the first
question on the supplementary forms must be linked to his or her responses on the MHSIP/YSS, so that we
can create the two groups. The PSS/FSS Supplementary Forms in the Appendix include a text box at the
bottom for the CSI County Client Number, which is also used on the MHSIP/YSS.
If you have FSS who work with TAY and their families, you may wish to include the YSS-Parent/Caregiver
and the YSS-Family FSS Supplement in your data collection plan. However, this survey is optional; if you
27
believe that few caregivers of TAY will take the YSS-Family survey because they do not normally accompany
their TAY to receive mental health services, you can exclude this survey from your evaluation design. If you
choose not to use the YSS-Parent/Caregivers and the FSS supplement, ensure that your evaluation plan
reflects this.
Using other measures: If your organization collects other consumer outcome measures regularly, you may
be able to add them to your evaluation. Any additional outcome data that can be linked to the MHSIP/YSS
respondents through a common identification number can be merged into the MHSIP/YSS dataset for
analysis. Data collected within up to three months after the time of the MHSIP/YSS is best. This ensures that
based on the responses to the MHSIP/YSS supplementary forms, the evaluation team will know which TAY
saw a PSS/FSS and be able to compare outcomes based on this distinction. In the original evaluation
conducted by San Diego County, additional outcome indicators were selected from measures that are
collected regularly from consumers. For adults, these included the IMR (Illness Management and Recovery
scale, measuring indicators of illness management) and the RMQ (Recovery Markers Questionnaire,
measuring recovery) for adults. For children/youth, CAMS (Child and Adolescent Measurement System,
measuring child and youth internalizing behaviors, externalizing behaviors, and social competence) and
CFARS (Child Functional Assessment Rating Scale, measuring multiple domains of functioning) were used.
If your organization faces challenges collection the MHSIP/YSS from TAY, you may wish to replace those
measures with different measures of outcomes and/or satisfaction with services.
Participant and Staff Burden: When considering other measures, please note that the more measures you
add to an evaluation, the more burdensome the project will become. Just because something might be
interesting to measure does not mean it should be included in the evaluation. Carefully consider participant
and staff burden when selecting measures, as not only will more measures be more difficult for the
consumers who take the surveys, but staff will have to analyze more data.

Sample Text
Our evaluation design uses data collection tools (i.e., surveys) to assess areas of interest in regards to process
and outcomes of TAY PSS/FSS services. This section describes the data collected, source, validity and
reliability, and data collection methods for the process and outcomes evaluations. Information is also included
about specific parts of the data collection tools that represent exactly how each process or outcome will be
measured, which are also known as indicators.

PROCESS EVALUATION MEASURES AND INDICATORS

Program Manager and TAY PSS/FSS Online Surveys

Data Description and Source
The Program Manager Survey collects data regarding usage/wages of PSS/FSS, perceptions of PSS/FSS
roles, training for PSS/FSS, perceptions of cost effectiveness, PSS/FSS hiring experiences/plan, PSS/FSS
certifications, consumer eligibility for PSS/FSS services, and successes/limitations/ challenges of PSS/FSS. The
TAY PSS/FSS Survey collects data regarding types of peer/family support services provided, frequency of each
type of service provision, perceptions/advantages/disadvantages of working as a PSS/FSS, trainings
completed, wages and hours worked, interest in career advancement within BHS, suggestions for improvement,
gaps in services, personal impact of working as a PSS/FSS, and relationships with non PSS/FSS staff.
These data will be obtained through two web-surveys for Program Managers and PSS/FSS. These surveys
were originally developed by San Diego County BHS. The Program Manager and PSS/FSS surveys have been
included in the Appendix.

Validity/Reliability
These forms have not been assessed for validity or reliability, but were developed using stakeholder feedback
and successfully implemented by San Diego County for a 2015 evaluation project.

Data Collection Methods
In [month, year], the evaluation team will administer these surveys via email to obtain feedback on TAY
PSS/FSS services from program managers and PSS/FSSs. The surveys will be sent to all program managers
28
system wide via email, who will be asked to distribute the email to PSS/FSSs in their respective programs
(including Support Specialists who work at their facilities but are employed by another agency). The surveys will
be significantly shorter for Program Managers and PSS/FSS who do not work with TAY, but sending the survey
to all programs may help your organization get a better sense of what proportion of PSS/FSS work with TAY vs.
other age groups. The surveys may be modified to target only TAY program managers and TAY PSS/FSS.

Process Evaluation Aim 1 Indicators
To address the Process Evaluation Aim (PE1), we identified the indicators described in the following list that
would support our analysis and interpretation of the outcomes data and assist in developing a plan to implement
quality improvement changes. Data to measure these indicators is collected in the Program Manager and
PSS/FSS Surveys (see Appendix for measures).

Program Manager Survey Indicators
1. Number of PSS/FSS in general and number of PSS/FSS who work with TAY and their families
2. PSS/FSS training/certification status and needs
3. Hiring experiences/plan for PSS/FSS
4. Options for/barriers to PSS/FSS job advancement
5. Client eligibility for PSS/FSS services
6. Successes/limitations/ challenges of PSS/FSS

PSS/FSS Survey Indicators

1. Types of services needed and the frequency they are provided to TAY/TAY caregivers by PSS/FSS
2. PSS/FSS wages
3. PSS/FSS job role, satisfaction, challenges
4. PSS/FSS training/certification status and needs
5. Options for/barriers to PSS/FSS job advancement
6. Gaps in PSS/FSS services
7. Impact of working as a PSS/FSS on PSS/FSS recovery

OUTCOMES EVALUATION MEASURES AND INDICATORS

MHSIP, YSS-Youth, YSS-Family, and Supplementary PSS/FSS Surveys
Data Description and Source
General satisfaction and consumer-reported outcomes data will come from eight domains of the current adult
MHSIP Consumer Survey and the Youth Services Survey (YSS), which has two versions; the YSS-Youth
forages 13-17, and the YSS-Family for caregivers of child consumers. The current surveys are available here:
http://www.cibhs.org/consumer-perception-surveys. Supplementary one-page surveys will be added to the
beginning of the MHSIP , the YSS-Youth completed by TAY, and the YSS-Family completed by TAY caregivers
to ask whether consumers/caregivers had interacted with a PSS/FSS in the course of their treatment, and if they
did, to answer satisfaction and perceived outcomes questions about their PSS/FSS interaction. This
supplemental measure was developed by San Diego County BHS. Copies of the MHSIP, YSS-Youth, and YSS-
Family supplemental measures are available in the Appendix.

Validity/Reliability
MHSIP/YSS: The MHSIP and YSS are implemented twice each year across California, and have been tested
across Californias diverse population of mental health service consumers. The MHSIP performs more strongly
than the YSS, but both break down into more or less the same factors and show good overall reliability and
16,17
validity.
The supplemental PSS/FSS forms were developed using stakeholder feedback and successfully
implemented with the MHSIP and YSS by San Diego County for a 2015 project. They have not been tested for
validity or reliability, but were developed collaboratively with researchers, administrators, providers, and age
appropriate consumers.

Data Collection Methods
In [Month, Year], [organization name] will supervise the administration of the MHSIP, YSS-Youth, and YSS-
Family to all consumers who sought services in community-based programs over a period of one week per
State procedures on the collection of these surveys. The supplementary forms will be added to the beginning of
16
Carlson, J. A., Sarkin, A. J., Levack, A. E., Sklar, M., Tally, S. R., Gilmer, T. P., & Groessl, E. J. (2011). Evaluating a measure of
social health derived from two mental health recovery measures: The California Quality of Life (CA-QOL) and Mental Health Statistics
Improvement Program Consumer Survey (MHSIP). Community Mental Health Journal, 47(4), 454-462.
17
Shafer, A. B., & Temple, J. M. (2013). Factor Structure of the Mental Health Statistics Improvement Program (MHSIP) Family and
Youth Satisfaction Surveys. The Journal of Behavioral Health Services & Research, 40(3), 306316. http://doi.org/10.1007/s11414-013-
9332-4
29
each survey for TAY consumers age 16-25 and TAY caregivers. The surveys will be mailed to programs for
administration in hard copy to their respective consumers throughout the week. The surveys and supplemental
forms will subsequently be collected by [organization name] for data entry prior to scanning the forms and
sending the hard copies of the MHSIP and YSS surveys on to the county/state. The supplemental PSS/FSS
forms will not be sent to the state, as they are not part of the standardized surveys and are only intended to be
used for this evaluation project. The MHSIP/YSS and the Supplementary forms all have a box for the CSI
County Client Number participant ID number that allows the data to be linked (or merged) together for analysis.
This is necessary to be able to compare TAY who have seen a PSS/FSS to those who have not, as a
participants answer to the first question on the supplementary forms must be linked to his or her responses on
the MHSIP/YSS so we can create the two comparison groups.

Outcomes Evaluation Aims 1 and 2 Indicators
To address the outcomes evaluation aims, we identified the following consumer satisfaction and outcome
indicators from the previously described measures. These indicators are drawn from the MHSIP Consumer
Survey (MHSIP), the Youth Services Survey (YSS), and supplementary forms added to the MHSIP and YSS
designed originally for an evaluation project in San Diego County in 2015 to assess consumer satisfaction and
outcomes from interacting with PSS/FSS. These measures are available in the Appendix (MHSIP and PSS/FSS
Supplement) and YSS-Youth & PSS/FSS Supplement).

Adult MHSIP Consumer Satisfaction Survey Indicators: The following list of indicators comes from the adult
MHSIP Consumer Survey. Adult TAY consumers will be asked to answer questions which cover eight domains
of consumer satisfaction and perceived outcomes, listed below. The MHSIP Consumer Survey includes
additional questions that include questions about quality of life, education, employment, housing, criminal justice
contacts, and socio-demographics (e.g., age, gender, race/ethnicity). See Appendix Tables 2 and 3 for question
numbers and text for each of the eight domains listed here.

MHSIP Outcome Indicators (Dependent Variables)
1. General Satisfaction
2. Perception of Access
3. Perception of Quality and Appropriateness
4. Perception of Outcomes
5. Functioning
6. Social Connectedness
7. Perception of Participation in Treatment Planning
8. Arrests/Police Interactions

Primary Stratifying Variable (Independent Variable)
- PSS interaction during care (MHSIP Supplementary PSS Form, Q1)

Other MHSIP Data (Control or Stratifying Variables)
- Demographics
o Age (Date of Birth)
o Gender
o Race/Ethnicity
o Duration in care
Youth Services Satisfaction Survey Indicators are drawn from the Youth Services Survey (YSS). There are
two versions of this survey; twice each year, the YSS-Youth is administered to all youth consumers age 13 and
older and the YSS-Family is administered to all available caregivers who receive services during the designated
18
data collection period. This evaluation study will use data from both versions of the YSS, which include
satisfaction and outcomes questions, and echoes the domains found in the adult MHSIP Satisfaction Survey.
However, the questions are slightly different across the three surveys to be relevant and phrased appropriately
for each group. See Appendix Tables 2 and 3 for question numbers and text for each of the eight domains listed
here.

YSS-Youth & YSS Family Outcome Indicators (Dependent Variables)
1. Satisfaction with Services
2. Participation in Treatment
3. Good Access to Service
4. Cultural Sensitivity
18
See MHSIP/YSS Instructions here: http://www.cibhs.org/consumer-perception-surveys
30
5.
6.
7.
8.
Positive Outcomes of Services

Functioning
Social Connectedness
Arrests/Police Interactions

Primary Stratifying Variable (Independent Variable)
- PSS/FSS interaction during care (YSS-Youth/YSS-Family Supplementary PSS/FSS Form, Q1)

Other YSS-Youth & YSS Family Data (Independent/Control Variables or Stratifying Variables)
- Demographics
o Age (Date of Birth)
o Gender
o Race/Ethnicity
o Duration in care

PSS/FSS Services Outcomes and Satisfaction will be measured via three supplemental one-page surveys
originally developed by San Diego County and added to the beginning of the MHSIP, the YSS-Youth, and the
YSS-Family implemented with TAY age 16-25 or caregivers of TAY. This measure was designed to be brief to
reduce consumer burden and to be easily added on to the MHSIP and the YSS. These supplementary forms
include measures which assess what services were provided to TAY by PSS/FSS, and perceived
outcomes/satisfaction from these PSS/FSS services. These data will provide more detailed information about
TAY consumer satisfaction and perceived outcomes from those TAY/caregivers who interacted with a
PSS/FSS.

2e. Data Entry/Management Plan
This section also includes a description of the data entry/management plan, including how the data were
entered or converted into a usable dataset. Keep all paper and scanned copies of completed surveys in a
secure environment, such as in a locked cabinet or in password protected files. Ensure that only relevant
staff have access to these files for evaluation or mandatory reporting purposes to protect the confidentiality
of the consumers.
Process Evaluation Data
We recommend using an online, web-based survey platform to collect the program manager and
PSS/FSS surveys. These tools vary in cost, ease of use, and security, including products such as Google
Forms (free, easy to use, less secure), SurveyMonkey ($30-100 monthly subscription, easy to use, more
secure), or Qualtrics ($1500-$2500 annual subscription, more difficult to use, more secure). SurveyMonkey
and Qualtrics have versions that meet HIPAA requirements for confidentiality of sensitive information;
although these precautions may not be necessary for evaluation surveys, steps should be taken to protect
the confidentiality of user responses. Online survey platforms eliminate the need for further data entry of
survey responses, as the responses are automatically entered into a dataset, and can usually be exported in
multiple formats for analysis. Many of these platforms have basic analysis and reporting functions as well.
An online survey platform may be the best option for the program manager and PSS surveys.

If an online survey platform is not an option for your organization, the surveys may be distributed in MS word
format via email or in paper form for completion by program managers and PSS/FSS, and then emailed or
mailed back to the evaluation team for data entry and analysis. For this option, use the supplementary
Double Data Entry Form for data entry.

31
Outcomes Evaluation Data

Obtaining datasets from the MHSIP/YSS is a challenge for many counties and providers. The California
19
State Department of Health Care Services (DHCS) uses a software called TeleForm to scan mandated
surveys and create datasets for analysis. This software is similar to Scantron, which has been used in the
past for standardized testing and other large-scale data collection activities. Generally, counties/providers
have a few options in regards to sending their mandated MHSIP/YSS survey data to the State.
Counties/providers can either send all hard copies or scanned digital copies of the completed surveys to the
county/state. The state then uses TeleForm to extract the data from the scans. Counties/providers often do
not get any data back for their own analysis using this option, especially those that do not create scanned
versions and send the hard copies of the surveys off to the state. If your organization does not receive
usable MHSIP/YSS data using these methods in Excel, comma-separated values file (.csv), or another type
of database format, or does not receive the data in a timely fashion, this is not an option for the evaluation.
We recommend using the following strategy to obtain usable MHSIP/YSS datasets: Remove the
PSS/FSS supplemental pages from the completed MHSIP/YSS for TAY, digitally scan or copy all surveys
locally and send the hard copies or digital files to the county/State for processing and to meet mandated
requirements. Then, use the copied/scanned files to enter the TAY MHSIP/YSS data and the supplemental
PSS/FSS surveys manually using the supplementary Double Data Entry Excel form or using TeleForm
software.
Using Excel for Data Entry

We recommend scanning the MHSIP/YSS surveys, sending the hard copies or the scanned files to the state,
and then having organization staff enter the data manually from the scanned TAY MHSIP/YSS files and the
supplementary PSS/FSS forms to create usable datasets for the evaluation. This is also time and labor-
intensive, but avoids the high cost of a TeleForm license and the technical skills required to program
TeleForm to read the supplemental surveys (see description of TeleForm below). In addition, for the
purposes of the evaluation, only TAY surveys need to be entered, although you may wish to enter all
completed MHSIP/YSS forms for other evaluation/quality improvement purposes. This toolkit includes a
Microsoft Excel spreadsheet template to facilitate manual data entry.
Use three separate copies of this template to enter the data from 1) the MHSIP and MHSIP PSS
Supplementary Form, 2) the YSS-Youth and YSS-Youth PSS Supplementary Form, and 3) the YSS-Family
and YSS-Family FSS Supplementary Form to create three datasets. The template includes instructions, and
incorporates a data quality check (i.e., double data entry) to ensure manually entered data is accurate. Once
any participant data is added to this form, it must be password protected (i.e., encrypted) to ensure
consumer confidentiality. The person conducting data entry should work with the person who will conduct the
analysis to ensure the data is entered in an easy-to-use format. For example, the analyst may instruct the
data entry staff to enter responses numerically, such as Strongly Disagree=1, Disagree=2, I am Neutral=3,
Agree=4, and Strongly Agree=5, enter all responses that were left blank as missing data, which can be
indicated by leaving a cell empty, and enter 99 for responses of not applicable. Record a list of codes used
for each question in the survey, which is also known as creating a codebook, so that the analyst can
correctly interpret the data.
Using TeleForm for Data Entry

To use TeleForm, each page of a blank survey is scanned at a high resolution and uploaded to the software
program. Then, the user creates a program that tells what answer each location on the page represents.
19
For more information on TeleForm:

Product website- http://www.electricpaper.biz/products/teleform.html
User brochure- http://www.electricpaper.biz/fileadmin/user_upload/downloads/eForm_Suite/eForm_Suite_Brochure.pdf
32
After this program is created, each completed survey is scanned and uploaded, then TeleForm reads each
20
answer and creates a database. The state provides TeleForm files for each round of the MHSIP and YSS.
Advantages of Using TeleForm: TeleForm gives counties/providers access to datasets to analyze for
evaluation and quality improvement purposes, and can saves time and labor costs of doing manual data
entry.
Disadvantages of Using TeleForm: TeleForm costs several thousand dollars for a single perpetual desktop
license, which may be prohibitively costly for some counties/providers. In addition, scanning each survey into
high resolution files can be time and labor-intensive. Finally, using TeleForm and creating new forms in
TeleForm may require additional staff training.
In San Diego County, UCSDs Health Services Research Center was contracted to scan the MHSIP/YSS
surveys and uploads them to the TeleForm software to create databases. The supplemental PSS/FSS
pages added to the MHSIP/YSS were either scanned using TeleForm or hand-entered into a separate
database. The supplemental dataset was then merged into the MHSIP/YSS dataset for analysis. In another
example from California, Los Angeles Countys Public Health Department has developed a system to use
TeleForm for efficient data collection in partnership with Hewlett Packards (HP) Autonomy software
21
platform. This flexible system is used to collect important data from multiple sources on local HIV/sexually
transmitted disease care and services.
We only recommend using TeleForm for the evaluation if your county/organization is already using
TeleForm for other projects, as the cost and training involved with setting up a TeleForm system is
considerable.

Sample Text
Data Security and Consumer Confidentiality
All paper and scanned copies of completed surveys will be kept in a secure environment, such as in a locked
cabinet or in password protected files. Only relevant staff will have access to these files for evaluation or
mandatory reporting purposes to protect the confidentiality of the consumers.

Process Evaluation
Data collected for the process evaluation (i.e., program manager and TAY PSS/FSS surveys) will be
downloaded from the online survey platform for analysis. No additional data entry processes will be needed.

Outcomes Evaluation
Data collected for the outcomes evaluation (i.e., MHSIP, YSS-Youth, YSS-Family, and the Supplementary
PSS/FSS forms for each of those surveys) will be scanned prior to sending the MHSIP/YSS surveys on to the
county/state. The supplementary PSS/FSS forms will not be sent to the state, as they are not part of the
mandated state data collection. These scanned forms will be entered manually into three copies of an excel
data entry template by [organization name] staff. The first copy will include the MHSIP and the MHSIP PSS
Supplementary form, the second copy will include the YSS-Youth and the YSS PSS Supplementary form, and
the third copy will include the YSS-Family and the YSS-Family FSS Supplementary form. This data entry
template includes features that allow double data entry, a process which requires the data to be entered twice
to identify and correct data entry errors. Discrepancies in the two datasets will be resolved by referring to the
original paper surveys. Once any consumer data is added to this form, it will be password protected (i.e.,
encrypted) to ensure consumer confidentiality.

20
TeleForm files for California are generally available from the California Institute for Behavioral Health Solutions
(CIBHS) website: http://www.cibhs.org/consumer-perception-surveys
21
HP Autonomy (2013) Case Study: Los Angeles Public Health HIV/STD Program utilizes HP Autonomy TeleForm
software to eliminate data entry. http://www.ndm.net/archiving/pdf/TeleForm%20-
%20Los%20Angeles%20Public%20Health.pdf
33
Section 3: Data Analysis

To help users with this section of the toolkit, we strongly recommend reading the Suicide Prevention
22
Evaluation Toolkit created by RAND Corporation for a detailed but user-friendly explanation of how to
conduct data analysis in MS Excel in the context of mental health program evaluation.

3a. Process Evaluation Data Analysis Plan (Program Manager and PSS/FSS Surveys)
Description
The analysis for the process evaluation is fairly simple for numerical data, involving the calculation of basic
descriptive statistics (e.g., means, frequencies/percentages) and inputting the results into the table templates
(Table Templates 1 & 2) provided in the Data Analysis and Reporting Section. These tables have row and
column headers, but do not include data, and can give you a better idea of how to report your findings.

Open-Ended Questions: This section also has placeholders above and below each table template for the
written narratives of the findings and summaries of the open-ended (text answer) questions found in the
surveys. To save time, you may wish to omit some of these open-ended questions from the analysis if they
are not of interest to your organization or are not directly relevant to your quantitative findings in the tables.
To analyze this qualitative data, you may wish to code each response into categories and count the number
of responses in each category. For each question you wish to analyze, read through the responses and list
categories that encompass the responses. For example, you can develop categories for the responses to
question 22 of the PSS/FSS Survey, what new programs for TAY and/or their families would you like to see
started if there were resources available? For this question, you might identify categories such as housing,
social support, education, and employment, and specific TAY groups who need these services based
on the responses that PSS/FSS have given. One way to organize your analysis is by using an Excel
spreadsheet. The open-ended question text for a question would go in the first column, and then each of
your categories would have labels at the top of the next columns. See the fictional example table below:

1
2
3
4
A
B
C
D
E
F
Open ended question: What new programs for TAY and/or
their families would you like to see started if there were
Social
Specific TAY
resources available?
Housing Support Education Employment groups in need
Our TAY need supported housing programs, especially our TAY
1

1
with children
The TAY I see need more opportunities to socialize and hang

1

out with other kids like them, they could all support each other.
I would really like to see education and employment programs
that target homeless or housing-insecure youth, maybe within a
1

1
1
1
program that also provides housing.
5 TOTAL
6 Percent (out of 3 responses)
2
67%
1
33%
1
33%
1
33%
2
67%

Using this table, you can then go through each text response and mark a 1 under each category that
the text mentions, as has been done in the example. You may add more codes as you go through the
text responses as they occur to you. The codes can also be as broad or as detailed as you want; for
example, instead of having just one column for special TAY populations, you could list each type of TAY
population in separate columns (e.g., homeless/housing insecure and TAY with children), and get a
sense of what TAY population is most in need. After you have gone through all the responses and
coded them with 1, you can use Excel formulas to calculate the total number of responses that
mentioned each category (using sum) and the percent of respondents who mentioned each category,
22
Acosta JD, Ramchand R, Becker A, Felton A, Kofner A. (2013). Suicide Prevention Evaluation Toolkit. RAND Corporation, Santa
Monica CA. Available at: http://www.rand.org/pubs/tools/TL111.html.
34
by dividing the totals in each column by the total number of responses overall. These calculations can
then be incorporated into the text of your report, along with any especially well-phrased quotes. You do
not have to report on all of the categories; those that were mentioned only rarely can be omitted (e.g.,
those mentioned by fewer than 5% of participants). Use this same process to analyze open-ended
questions in the Outcomes Evaluation part of the project.

Sample Text
Descriptive statistics (i.e., means, percentages) will be prepared for data from the Program Manager Surveys
and PSS/FSS Surveys. Scores will be used for some questions; for example, if a question or series of questions
were answered on a five point Likert scale, a score of 1-5 will be reported. These data may be stratified by type
of program (e.g., counseling, support, education) or other characteristics of interest.

Responses to open ended questions where the respondent has written their own text as a response will be
summarized and included in the report. A set of codes that correspond to topic areas or categories will be
developed based on an initial reading of the text responses for each question that is analyzed. Then, each text
response will be coded to indicate if it mentions each of the categories. Counts/frequencies of coded text will be
presented in the report, along with representative quotes.

3b. Outcomes Evaluation Data Analysis Plan (MHSIP/YSS-Youth/YSS-Family and Supplementary
PSS/FSS Forms)
When San Diego County implemented this evaluation, their analysts used a statistical analysis approach
called a two-way Analysis of Variance (i.e., ANOVA or sometimes known as ANCOVA if covariates are used)
to compare mean outcomes between groups of TAY or TAY caregivers. Similar to the Process Evaluation
data analysis section, the toolkit also includes table templates in the Data Analysis and Reporting Section for
the Outcomes Evaluation data, which you can fill in as the analysis is completed.
For this evaluation, the analyst will first provide descriptive demographic statistics for TAY who completed
the MHSIP, the YSS-Youth, and the YSS-Family, such as mean age, the percentage of males vs. females,
and racial/ethnic percentages (Table Templates 3, 4, and 5). Then, instead of analyzing each outcome
question in the MHSIP/YSS individually, the analyst can create score variables for each domain in the
MHSIP/YSS-Youth/YSS-Family that reflect a participants responses to a set of related questions. For detail
on which questions should be used in the analysis and which questions belong in each domain, see
Appendix Tables 1 (MHSIP), 2 (YSS-Youth), and 3 (YSS-Family). To create numeric score variables for each
domain, the analyst has to assign each response a number. This is also known as recoding the data. For
example, the analyst would recode the responses to each question in the General Satisfaction domain of
the MHSIP with numbers in ascending order (Table 1, Appendix): Strongly Disagree=1, Disagree=2, I am
Neutral=3, Agree=4, and Strongly Agree=5, and code all responses that were left blank or Not applicable to
(missing), which can be an empty cell or a cell with a period (.) in it. Then, the resulting recoded variables
would be summed and divided by five to create a new mean score variable, which would show each
respondents General Satisfaction score. This score variable can then be used for analysis.
After creating scores, the analyst could use ANOVA analyses to for each outcome indicator (e.g., mean
scores for functioning questions, satisfaction questions), comparing the following groups of TAY consumers
who took the MHSIP: (1) those who had contact with a PSS vs. (2) those who did not (Table Template 3). the
analyst would then conduct ANOVAs for each outcome variable comparing these two groups of TAY who
took the YSS-Youth (Table Template 4) and the TAY caregivers who took the YSS-Family (Table Template
5). These analyses can use participants duration of care as a control variable. This means that the
difference in outcomes that could be attributed to consumers duration of care will be incorporated (or
adjusted) into the final results, making the findings from the different groups more comparable. The analysts
can also stratify the results, meaning that they conducted separate analyses for different racial/ethnic groups
35
of consumers (Table Templates 6, 7, and 8), age groups, gender, or other relevant factors (create new table
templates similar to Tables 6, 7, and 8 if needed).
The data analysis plan for the PSS/FSS Supplementary forms involves calculating descriptive statistics
(e.g., means, frequencies), and inputting the findings into Table Templates 9, 10, and 11 for each of the
three forms. Again, there are placeholders in the Data Analysis and Reporting Section to write the narrative
of the findings displayed in the tables, and summarize any open-ended question responses.
Sample Text
Descriptive statistics will be provided (e.g., means/frequencies) for TAY consumers socio-demographic, usage,
duration of care, and outcome data. The two study groups (i.e., TAY that interacted with a PSS/FSS vs. those
that did not) will be compared using ANOVA (analysis of variance) tests to determine if there are any significant
differences in the selected outcomes (described above). Scores will be used for some questions; for example, if
a question or series of questions were answered on a four point Likert scale, a score of 1-4 will be reported.
Separate ANOVA analyses will be conducted for TAY who completed the MHSIP, TAY that completed the YSS-
Youth, and TAYs caregivers who completed the YSS-Family. We plan to control for duration of care in the
analyses, and present a stratified analysis of PSS/FSS exposure outcomes by groups of interest (e.g., younger
vs. older TAY, gender, race/ethnicity). Findings will be considered statistically significant at p<0.05.

Descriptive statistics will be provided for the data collected through the three PSS/FSS Supplementary forms.
Similar to the previously described analysis, some data may be presented using scores based on a questions
response categories, and these findings may be stratified by groups of interest.

As in the Process Evaluation part of this project, responses to open ended questions where the respondent has
written their own text as a response will be summarized and included in the report. A set of codes that
correspond to topic areas or categories will be developed based on an initial reading of the text responses for
each question that is analyzed. Then, each text response will be coded to indicate if it mentions each of the
categories. Counts/frequencies of coded text will be presented in the report, along with representative quotes.

Outcomes Evaluation Data Analysis Options

If you have someone in your organization with the statistical analysis skills available to conduct the
previously described analysis or another advanced analysis, please do so. If not, we are including a few
simpler but less rigorous ways to analyze the data. It is important for outcome evaluations to assess
statistical significance, which means you should use techniques that show whether your results are most
likely real or only due to chance. Presenting the means of two outcomes and saying that one is higher than
the other based on the observed difference between the two values is insufficient. Adapt the alternate
analysis plan text below for your own evaluation based on the analysis capabilities you have available.
Alternate Sample Text
Simple Alternative Analysis Plan- T-tests for Independent Samples (normally distributed continuous
data) or Wilcoxon Rank Sum Tests (non-normally distributed continuous data): Descriptive statistics (i.e.,
means, percentages) will be presented for consumers socio-demographic and outcome data. Each continuous
outcome variable will be assessed for normal distribution by creating histograms of the data or using an
appropriate statistical test for normality, such as the Shapiro-Wilk test. We will compare each outcome variable
between TAY/TAY caregivers who interacted with a PSS/FSS and those who did not using t-tests for two
independent samples (for normally-distributed continuous data) or Wilcoxon Rank Sum Tests (for non-normally
distributed continuous data). This analysis cannot account for covariates, although we may test relationships
with other relevant variables (e.g., t-test of mean MHSIP satisfaction score by gender) to determine if there are
other characteristics that could affect the outcomes. Differences in outcome variables will be considered
statistically significant at p<0.05.
Intermediate Alternative Analysis Plan- Linear Regression with Covariates: Descriptive statistics (i.e.,
means, percentages) will be presented for consumers socio-demographic and outcome data. Each continuous
outcome variable will be assessed for normal distribution by creating histograms of the data or using an
appropriate statistical test for normality, such as the Shapiro-Wilk test. We will conduct linear regression
analyses to assess the effect of interaction with a PSS/FSS (i.e., the independent variable) on each outcome
variable (i.e., dependent variables). Covariates including age, gender, race/ethnicity, or duration of care may be
included in the regression analyses. Regression coefficients will be considered statistically significant at p<0.05.
36
REPORTING THE FINDINGS

Report Template
This section provides an outline for a written report to share the findings of the TAY PSS/FSS Evaluation. It
includes a description of what the introduction and methods section of a report should contain, table
templates for displaying the results of the evaluation
Executive Summary
Although this section will be one of the first parts of the report, it should be written last, after all other sections
have been written. The Executive Summary should serve as a brief summary report, and include the same
sections as the larger full length report: Introduction, Methods, Results, and Discussion/Next Steps.
Summarize the most important points about the background and methodology used, the most striking
findings from the process and outcomes evaluations, and the most salient messages and recommendations
from the discussion.

Introduction & Methods

The introduction and methods section of the report provide important background information to help readers
understand the results of the evaluation. For this section of the report, summarize the evaluation plan text
regarding the introduction and methods. Change the verb tense of the methods from future tense (e.g., the
evaluation will collect surveys from TAY) to past tense (e.g., the evaluation collected surveys from TAY).
Keep these sections brief; if you need to add additional information, consider including an appendix with
greater detail on the background and methods.
Results
The results section of the report is where you describe the study findings. Fill in the table templates provided
below, which were described in the data analysis part of the Methods section, with the results of the data
analysis. The questions in each survey are numbered, and those question numbers are included referenced
in the tables. Replace the question numbers (e.g., Q1, Q2, Q3, etc.) with the mean averages or percentages
from your data. After filling the numbers in on the tables, add a narrative description of the data in the tables
before each table. In your writing about the data, present the results; do not try to simultaneously interpret
them, as that will be done in the discussion section. If necessary, the tables can be placed in an appendix
and referred to in the text of the report. You may wish to shorten or break up some of the templates into
smaller tables, or omit some of the open-ended questions from the analysis process or report if they are not
of interest to your organization.

37
Sample Text

PROCESS EVALUATION
Findings from the Program Manager Surveys

[Insert narrative of Program Manager Survey findings, including ranking of findings from best to worst
responses, using the data in Table 1]

Table Template 1: Responses from Program Manager Surveys of PSS/FSS programs for TAY [Month,
Year of Survey, Organization Name].
Program Manager Survey Questions

Behavioral Health Services System of Care (% of program managers working in
each system)
Children, Youth, & Families (CYF)
Adults & Older Adults (AOA)
Both CYF & AOA
Types of Services Provided (% of program managers indicating each type of
service provided)
Assertive Community Treatment (ACT)
Behavioral Health (BH) Court
Child and Adolescent Psychiatric Services (CAPS)
Case Management
Case Management- Institutional
Case Management- Strengths
Case Management- Transitional
Day Treatment
Fee for Service (FFS)
Juvenile Forensic Services (JFS)
Outpatient
Prevention (PEI)
Emergency Psychiatric Unit (EPU)
Crisis Residential
Residential
Therapeutic Behavioral Services
Inpatient Hospital
Other
Do you have Peer Support Specialists (PSS) or Family Support Specialists (FSS)
at your facility?
Yes, both PSS and FSS
PSS only
FSS only
No, we do not have PSS or FSS
Total number of PSS/FSS in program (mean)
Do you utilize PSS/FSS for TAY services at your facility? (% Yes)

Total number of TAY PSS/FSS in program (mean)

Functions of TAY PSS/FSS (% of program managers indicating each function)
Provide advice or counseling to consumers
Help consumers understand what resources are available
Help consumers fill out paperwork
Help consumers understand what staff is asking of them
Being a role model (for recovery)
Help to create/set recovery goals
Help with monitoring progress
Help with navigating the behavioral health services system
Provide social or emotional support
Administrative/clerical
All
Respondents
Mean/
Percentage
(total number of
respondents
[n]=XX)
-
Programs with
TAY PSS/FSS
Mean/
Percentage
(n=XX)
(answering
yes to Q9)
-
Question (Q) 1
Q1
Q1
-
Q1
Q1
Q1
-
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
-
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
Q4
-
Q5
Q5
Q5
Q5
Q7
Q8
Q5
Q5
Q5
Q5
Q7
Q8
Programs with
TAY PSS/FSS
Mean/
Percentage
(n=XX)
Q10
-
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
38
Help coordinating physician visits and other medical appointments

Arranging transportation to and from medical services
Accessing and maintaining insurance coverage
Providing education about medical conditions and recovery strategies
Facilitating communication with health care providers
Maintaining telephone contact between patients and health care providers
Motivate and educate consumers about the importance of preventive services
Coordinating care among providers
Arranging for translation services
Providing education to improve health literacy
Assisting with the financing and management of medication
Assisting with issues related to housing
Other
Program uses mostly paid vs. volunteer TAY PSS/FSS (% of program managers)
Paid
Volunteer
Are paid TAY PSS/FSS cost-effective? (% yes)
Do your paid PSS/FSS work full time, part time, both?
All PSS/FSS are full time
All PSS/FSS are part time
Some PSS/FSS are full time, some are part-time
Would you increase the wages of your paid TAY PSS/FSS if possible? (% yes)
Contracted or directly employed TAY PSS/FSS (% of program managers indicating contracted vs.
directly employed)
Contracted by another agency
Paid directly from program budget
Hiring plans for TAY PSS/FSS in the next 12 months (% of program managers)
Downsize by several (3+)
Downsize a little (1-2)
Stay with what we've got now
Increase a little (1-2)
Increase by several (3+)
Unsure
Other
Types of Training for TAY PSS (% of program managers)
Motivational Interviewing
PET - Peer Employment Training (through RICA)
WRAP - Wellness Recovery Action Planning
WELL - Wellness and Empowerment in Life and Living
Medication for Success
Transformational Advocacy
Advocacy for Positive Outcomes
Recovery Practices in Leadership and Coaching
Other trainings through BHETA (Behavioral Health Education and Training Academy)
Other
Types of Training for TAY FSS (% of program managers)
Other
Are your TAY PSS certified?
Yes, all are certified
Yes, some are certified
No, none are certified
Unsure/Dont know
Average caseload for TAY PSS per week
Average caseload for TAY FSS per week
Time per TAY PSS Session
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
-
Q12
Q12
Q13
-
Q14
Q14
Q14
Q15
-
Q16
Q16
-
Q17
Q17
Q17
Q17
Q17
Q17
Q17
-
Q18
Q18
Q18
Q18
Q18
Q18
Q18
Q18
Q18
Q18
-
Q19
Q19
Q19
Q19
Q19
Q19
Q19
Q19
Q19
Q19
-
Q20
Q20
Q20
Q20
Q23
Q24
-
39
0-15 minutes
16-30 minutes
31-45 minutes
46-60 minutes
61-90 minutes
Q25
Q25
Q25
Q25
Q25
90+ minutes
Time per TAY FSS Session
0-15 minutes
16-30 minutes
31-45 minutes
46-60 minutes
61-90 minutes
90+ minutes
How do TAY PSS/FSS usually work?
One-on-one
In groups as a lead or co-facilitator
In groups as a support to someone else who is facilitating
Other
How would you rate your experience finding and hiring TAY PSS/FSS
Very difficult
Somewhat difficult
Neutral
Somewhat easy
Very easy
How would you rate your experience in being able to retain TAY PSS/FSS staff?
Very difficult
Somewhat difficult
Neutral
Somewhat easy
Very easy
Have any TAY PSS/FSS advanced to a different job in the program? (% yes)
Q25
-
Q26
Q26
Q26
Q26
Q26
Q26
-
Q27
Q27
Q27
Q27
-
Q28
Q28
Q28
Q28
Q28
-
Q29
Q29
Q29
Q29
Q29
Q29

Summary of Program Manager Survey Open-Ended Questions:
Q2. Your job title/role?
Q4. Types of services your program provides? Other
Q6. If your program does not use the job title "Peer Support Specialist" and "Family Support Specialist," what
job title is used for the staff members that are required to have "lived experience"?
Q9. If your program does not use the job title "Peer Support Specialist" and "Family Support Specialist," what
job title is used for the staff members who work with TAY that are required to have "lived experience"?
Q11. Which functions do the PSS/FSS at your facility perform in providing services to TAY and/or their families?
Other
Q17. What are your hiring plans for PSSs/FSSs for TAY and/or TAYs families in the next 12 months? Other
Q18. What kind of training do your PSSs who work with TAY typically have? Other
Q19. What kind of training do your FSSs who work with TAY and/or their families typically have? Other
Q22. From where do you typically recruit or hire your PSSs and FSSs who work with TAY and/or TAYs
families?
Q27. How do the PSSs/FSSs who work with TAY and/or TAYs families at your facility typically work? (other):
Q30. What QUALIFIES or DISQUALIFIES a TAY consumer/caregiver as someone who gets assistance from a
PSS/FSS?
Q31. Are there any drawbacks or limitations in having PSSs and/or FSSs who work with TAY and/or TAYs
families at your program?
Q33. What advancement possibilities do you see for PSSs/FSSs who work with TAY and/or TAYs families in
your program?
Q34. Do you have any additional comments that you'd like to share with us about your or your program's
experience with PSSs or FSSs who work with TAY and/or TAYs families?

40
Findings from the PSS/FSS Surveys

[Insert narrative of PSS/FSS Survey findings, including ranking of findings from best to worst responses, using
the data in Table 2]

Table Template 2: Responses from PSS/FSS Surveys [Month, Year of Survey, Organization Name].
PSS/FSS Survey Questions

Behavioral Health Services System of Care (% of PSS/FSS
working in each system)
Children, Youth, & Families (CYF)
Adults & Older Adults (AOA)
Both CYF & AOA
Type of PSS/FSS
PSS: Current or former consumer of mental health services or
an individual with lived experience with mental health
challenges.
FSS: Parent or caregiver of a current or former consumer of
mental health services
Parent/caregiver of youth (17 and under)
Parent/caregiver of adult (18 and older)
FSS: Non-parental family member of a current or former
consumer of mental health services (e.g., spouse, sibling, etc.)
Non-parental family member of youth (17 and under)
Non-parental family member of adult (18 and older)
FSS: Other (e.g., friend, neighbor, etc.) (FSS)
% of PSS/FSS indicating they work with the following
populations:
Transition Age Youth (TAY) only (services targeted specifically
towards those in the TAY age ranges of 16 through 25) and
NOT their families/caregivers
TAY and/or families/caregivers of TAY (services targeted
specifically towards those in the TAY age ranges of 16
through 25)
Children (ages 0-5) and/or their families/caregivers
Older children and adolescents (ages 6-15) and/or their
families/caregivers
Adult and Older Adult (ages 25+)
Older Adult specific services (ages 60+)
All PSS/FSS
Mean/
Percentage
(n=XX)
-
TAY PSS
Mean/
Percentage
(n=XX)
(answered
consumer/
former
consumer in
Q2)
-
TAY FSS
Mean/
Percentage
(n=XX)
(answered
they were a
family
member in Q2)
-
Q1
Q1
Q1
-
Q2
Q1
Q1
Q1
-
Q2
Q1
Q1
Q1
-
Q2
Q2
Q2
Q2
Q2
Q2
Q2
Q2
Q2
Q2
Q2
Q2
Q2
Q2
Q2
Q2
-
Q2
Q2
Q2
-
Q2
Q2
Q2
-
Q3
Q3
Q3
Q3
Q3
Q3
Q3
Q3
Q3
Q3
Q3
Q3
Q3
Q3
Q3
Q3
TAY PSS
Mean/
Percentage
(n=XX)
(Answered that
they worked
with TAY in Q3)
-
Q3
Q3
TAY FSS
Mean/
Percentage
(n=XX)
(Answered
that they
worked with
TAY in Q3)
-
Q4a
Q4b
Q4a
Q4b
Q4c
Q4d
Q4c
Q4d
Q4e
Q4f
Q4e
Q4f
Q4g
Q4g
Q4h
Q4h
Q4i
Q4i
Need and Provision of Services Provided to TAY/Families by PSS/FSS

Scoring: 1=Never, 2=Sometimes, 3=Often, 4=Always
How often do the TAY consumers or families you work with NEED this type of
service?
Coordinating physician visits and/or other mental or physical health appointments.
Arranging transportation to and from mental or physical health services.
Accessing and maintaining insurance coverage.
Providing education about mental health problems and recovery/management
strategies.
Facilitating communication with mental health care providers.
Maintaining telephone contact between consumers and mental health care
providers.
Motivating and educating individuals or their family/caregivers about the
importance of preventive services.
Assisting individuals/families/caregivers in completing medical, financial, and other
forms.
Coordinating care among providers.
41
Arranging for translation services.

Providing education to improve mental health literacy.
Providing emotional support.
Assisting with medication management and financing.
Assisting with issues related to housing.
Assisting with issues related to employment.
How often do you typically PROVIDE this type of service in your role as a
Peer/Family Support Specialist?
Coordinating physician visits and/or other mental or physical health appointments.
Arranging transportation to and from mental or physical health services.
Providing education about mental health problems and recovery/management
strategies.
Facilitating communication with mental health care providers.
Maintaining telephone contact between consumers and mental health care
providers.
Motivating and educating individuals or their family/caregivers about the
importance of preventive services.
Assisting individuals/families/caregivers in completing medical, financial, and other
forms.
Providing education to improve mental health literacy.
Providing emotional support.
Assisting with medication management and financing.
Assisting with issues related to employment.
What is it like to work as a Peer/Family Support Specialist? (Scoring: 1=Strongly
disagree, 2=Somewhat disagree, 3=Somewhat agree, 4=Strongly agree)
I have a clear job description.
I am clear about what I can and cannot do in my role as a Peer/Family Support
Specialist.
Identifying as both a consumer and a staff member is challenging for me.
I identify with the consumers more than with other staff.
I receive high quality supervision.
I receive enough supervision.
I receive the individual support I need.
I am afraid to ask for help.
I feel comfortable discussing my diagnosis with others.
I experience burnout.
I experience feelings of isolation in my role as Peer/Family Support Specialist.
I get paid an adequate amount for the services I provide.
I experience benefits from interacting with consumers.
I am recognized as a valuable member of the team by the non-Peer/Family
Support Specialist staff.
I feel stigmatized by the non-Peer/Family Support Specialist staff.
I think my presence here benefits the other staff.
I think I am a positive role model of a consumer in recovery for the non-
Peer/Family Support Specialist staff.
It seems like the non-Peer/Family Support Specialist staff do not like mental health
consumers.
I have good communication with other staff.
I feel like a colleague with the other staff.
The culture where I work is Peer/Family Support Specialist friendly.
PSS/FSS Trainings Completed
PET Peer Employment Training (through RICA)
WRAP Wellness Recovery Action Planning
WELL Wellness and Empowerment in Life and Living
Other trainings through BHETA (Behavioral Health Education and Training
Academy)
Other
How would you rate your experience finding a job as a PSS/FSS?
Very difficult
Q4j
Q4k
Q4l
Q4m
Q4n
Q4o

Q5a
Q5b
Q5c
Q5d
Q4j
Q4k
Q4l
Q4m
Q4n
Q4o

Q5a
Q5b
Q5c
Q5d
Q5e
Q5f
Q5e
Q5f
Q5g
Q5g
Q5h
Q5h
Q5i
Q5j
Q5k
Q5l
Q5m
Q5n
Q5o
-
Q5i
Q5j
Q5k
Q5l
Q5m
Q5n
Q5o
-
Q6a
Q6b
Q6a
Q6b
Q6c
Q6d
Q6e
Q6f
Q6g
Q6h
Q6i
Q6j
Q6k
Q6l
Q6m
Q6n
Q6c
Q6d
Q6e
Q6f
Q6g
Q6h
Q6i
Q6j
Q6k
Q6l
Q6m
Q6n
Q6o
Q6p
Q6q
Q6o
Q6p
Q6q
Q6r
Q6r
Q6s
Q6t
Q6u
-
Q7
Q7
Q7
Q7
Q7
Q7
Q7
Q7
Q7
Q6s
Q6t
Q6u
-
Q7
Q7
Q7
Q7
Q7
Q7
Q7
Q7
Q7
Q7
-
Q8
Q7
-
Q8
42
Somewhat difficult
Neutral
Somewhat easy
Very easy
When you meet with consumers, where are your sessions most typically held?
In a designated private office or room
In any available private office or room
In the waiting room of the clinic or hospital
At a designated desk in a larger shared common room
In a shared common room
In a hallway or corridor
Other
Mean per-hour wage working as a PSS/FSS ($)
Hours per week worked as a PSS/FSS
Less than 5 hours per week
5-10 hours per week
11-15 hours per week
More than 40 hours per week
If your program could offer you more hours, would you want them? (% yes)
Would you be concerned about losing your benefits if you worked more hours or
made more money?
Yes
No
N/A
Are you interested in advancing your career to another type of job within the
County of San Diego Behavioral Health Services? (% yes)
Has your perception of being a Peer/Family Support Specialist improved,
worsened, or remained the same since you started working as a Peer/Family
Support Specialist?
Improved
Worsened
Remained the same
Has working as a Peer/Family Support Specialist helped you with your own
recovery/treatment management?
Yes
No
Other
Q8
Q8
Q8
Q8
-
Q9
Q9
Q9
Q9
Q9
Q9
Q9
Q10
-
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q12
-
Q8
Q8
Q8
Q8
-
Q9
Q9
Q9
Q9
Q9
Q9
Q9
Q10
-
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q11
Q12
-
Q13
Q13
Q13
Q14
Q13
Q13
Q13
Q14
Q16
Q16
Q16
Q16
Q16
Q17
Q20
Q20
Q20
Q20
Q20
Q20

Summary of PSS/FSS Survey Open-Ended Questions:
Q2. Please choose the category that best describes you: Other
Q7. Which, if any, of these trainings have you completed? Other
Q9. When you meet with consumers, where are your sessions most typically held? Other
Q15. If so, please list the job title and please describe the training you feel would be appropriate to get you
there. [if participant said yes to 47. Are you interested in advancing your career to another type of job within
the County of San Diego Behavioral Health Services?]
Q17. What do you LIKE about being a Peer/Family Support Specialist for TAY and/or their families?
Q18. What do you DISLIKE about being a Peer/Family Support Specialist for TAY and/or their families, or
where/when you feel stress in this role?
Q19. How might your relationship with non-Peer/Family Support Specialist staff be improved? Do you have any
suggestions for improving the Peer/Family Support Specialist program for TAY and/or their families at your
facility and/or systemwide?
Q21. If yes, HOW has being a Peer/Family Support Specialist for TAY and/or their families helped affected your
own recovery/treatment management?
Q22. What new programs for TAY and/or their families would you like to see started if there were resources
available?
Q23. Where do you see yourself in 5 or 10 years within the Behavioral Health Services system of care?
Q24. Do you have any other feedback that you would like to provide about being a Peer/Family Support
Specialist for TAY and/or their families?

43
OUTCOMES EVALUATION FINDINGS

Findings from the MHSIP/YSS: Demographics, Outcomes, and Satisfaction Measures

[Insert narrative of MHSIP findings, including ranking of findings from best to worst responses, using the data
from Table 3]

Table Template 3: Demographics, Mean Outcomes and Satisfaction scores for all [Organization Name]
TAY participants who completed the MHSIP, and comparing TAY who interacted with a PSS to TAY who
did not interact with a PSS in the last six months [Month, Year of MHSIP Survey].
See Appendix Table 1 for Question Text

MHSIP Demographics
Mean age (min-max)
Gender
% Male
% Female
Ethnicity- % Mexican, Latino, Hispanic origin
Race
% American Indian/Alaska Native
% Asian
% Black/African American
% Native Hawaiian/Other Pacific Islander
% White/Caucasian
% Other
% Unknown
Duration in Care
First visit
More than one visit but receiving services for less than 1
month
1-2 months
3-4 months
6-12 months
1 year or more
MHSIP DOMAINS (See Appendix Table 1 for unlisted
question text and numbers)
General Satisfaction
Perception of Access
Perception of Quality and Appropriateness
Perception of Outcomes
Functioning
Perception of Participation in Treatment Planning
1
Significantly different values: *** p<0.01, ** p<0.05, * p<0.10
All TAY
Participants
(n=XX)
Page 5, Q11
-
Page 4, Q8
Page 4, Q8
Page 4, Q9
-
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
-
Page 4, Q1
Page 4, Q1
TAY who
interacted with a
1
PSS (n=XX)
Page 5, Q11
-
Page 4, Q8
Page 4, Q8
Page 4, Q9
-
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
-
Page 4, Q1
Page 4, Q1
TAY who did not

interact with a PSS
1
(n=XX)
Page 5, Q11
-
Page 4, Q8
Page 4, Q8
Page 4, Q9
-
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
Page 4, Q10
-
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Page 4, Q1
Mean scores, TAY
Mean scores, all
who interacted
Mean scores, TAY
TAY participants
with a PSS
who did not interact
1
1
(n=XX)
(n=XX)
with a PSS (n=XX)

Page 4, Q4 & Q7 Page 4, Q4 & Q7
Page 4, Q4 & Q7
Summary of TAY responses to MHSIP Survey Open-Ended Questions:

[Obtain relevant open-ended questions from current MHSIP, if any]

44
[Insert narrative of YSS-Youth findings, including ranking of findings from best to worst responses, using data
from Table 4]

Table Template 4: Demographics, Mean Outcomes and Satisfaction scores for all [Organization Name]
TAY participants who completed the YSS-Youth, and comparing TAY who interacted with a PSS to TAY
who did not interact with a PSS in the last six months [Month, Year of YSS-Youth Survey].

YSS-Youth Demographics
Mean age (min-max)
Gender
% Male
% Female
Ethnicity- % Mexican, Latino, Hispanic origin
Race
% Asian
% White/Caucasian
% Other
% Unknown
Duration in Care
First visit
More than one visit but receiving services for less than 1
month
1-2 months
3-4 months
6-12 months
1 year or more
YSS-Youth DOMAINS (See Appendix Table 2 for unlisted
question text and numbers)
Satisfaction with Services
Participation in Treatment
Good Access to Service
Cultural Sensitivity
Functioning
1
All TAY
Participants
(n=XX)
Page 4, Q20
-
Page 4, Q17
Page 4, Q17
Page 4, Q18
-
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
-
Page 2, Q4
Page 2, Q4
TAY who
interacted with a
1
PSS (n=XX)
Page 4, Q20
-
Page 4, Q17
Page 4, Q17
Page 4, Q18
-
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
-
Page 2, Q4
Page 2, Q4
TAY who did not

interact with a PSS
1
(n=XX)
Page 4, Q20
-
Page 4, Q17
Page 4, Q17
Page 4, Q18
-
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
Page 4, Q19
-
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Page 2, Q4
Mean scores, TAY
Mean scores, all
who interacted
Mean scores, TAY
TAY participants
with a PSS
who did not interact
1
1
(n=XX)
(n=XX)
with a PSS (n=XX)

Page 3, Q7 & Q13 Page 3, Q7 & Q13
Page 3, Q7 & Q13

Summary of TAY responses to YSS-Youth Survey Open-Ended Questions:
[Obtain relevant open-ended questions from current YSS-Youth, if any]
45

[Insert narrative of YSS-Family findings, including ranking of findings from best to worst responses, using data
from Table 5]

Table Template 5: Mean Outcomes and Satisfaction scores for all [Organization Name] Caregivers of
TAY participants who completed the YSS-Family, and comparing caregivers of TAY who interacted with
a FSS to those who did not interact with a FSS in the last six months [Month, Year of YSS-Family
Survey].

YSS-Family Demographics
Mean age of youth (min-max)
Youth gender
% Male
% Female
Youth Ethnicity- % Mexican, Latino, Hispanic
origin
Youth Race
% Asian
% White/Caucasian
% Other
% Unknown
Duration in Care
First visit
More than one visit but receiving services for
less than 1 month
1-2 months
3-4 months
6-12 months
1 year or more
All TAY
Caregivers
(n=XX)
Page 4, Q21
-
Page 4, Q18
Page 4, Q18
Page 4, Q19
TAY Caregivers who

interacted with a FSS
1
(n=XX)
Page 4, Q21
-
Page 4, Q18
Page 4, Q18
Page 4, Q19
TAY Caregivers who did

not interact with a FSS
1
(n=XX)
Page 4, Q21
-
Page 4, Q18
Page 4, Q18
Page 4, Q19
-
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
-
Page 2, Q5
Page 2, Q5
-
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
-
Page 2, Q5
Page 2, Q5
-
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
Page 4, Q20
-
Page 2, Q5
Page 2, Q5
Page 2, Q5
Page 2, Q5
Page 2, Q5
Page 2, Q5
Page 2, Q5
Page 2, Q5
Page 2, Q5
Page 2, Q5
Mean scores, TAY
Caregivers who
interacted with a FSS
1
(n=XX)

Page 3, Q8 & Q14
Page 2, Q5
Page 2, Q5
Page 2, Q5
Page 2, Q5
Mean scores, TAY
Caregivers who did not
interact with a FSS
1
(n=XX)

Page 3, Q8 & Q14
Mean scores, all

YSS-Family DOMAINS (See Appendix Table 3
TAY caregivers
for unlisted question text and numbers)
(n=XX)





Functioning


Page 3, Q8 & Q14
1

Summary of TAY caregivers responses to YSS-Family Survey Open-Ended Questions:
[Obtain relevant open-ended questions if any from current YSS-Family, if any]
46
Stratified MHSIP/YSS Findings

[Insert narrative of MHSIP Stratified findings, including ranking of findings from best to worst responses, based
on data from Table 6.]

Table Template 6: Mean Outcomes and Satisfaction scores for all [Organization Name] TAY participants
who completed the MHSIP, stratified by [gender, race/ethnicity, duration of care]. [Month, Year of MHSIP
Survey].
Mean scores, Mean scores, Mean scores, Mean scores, Mean scores,
MHSIP DOMAINS (See
Mean scores,
TAY who
TAY who
TAY who
TAY who
TAY who
Appendix Table 1 for
all TAY
identify as
identify as
identify as
identify as
identify as
unlisted question text and participants
_____
_____
_____
_____
_____
1
1
1
1
1
numbers)
(n=XX)
(n=XX)
(n=XX)
(n=XX)
(n=XX)
(n=XX)
General Satisfaction

Perception of Access

Perception of Quality and

Appropriateness
Perception of Outcomes

Functioning


Perception of Participation in

Treatment Planning

1
47
[Insert narrative of YSS-Youth Stratified findings, including ranking of findings from best to worst responses,
based on data from Table 7]

Table Template 7: Mean Outcomes and Satisfaction scores for all [Organization Name] TAY participants
who completed the YSS-Youth, stratified by [gender, race/ethnicity, duration of care]. [Month, Year of
YSS-Youth Survey].
YSS-Youth DOMAINS (See Mean scores,
TAY who
TAY who
TAY who
TAY who
TAY who
Appendix Table 2 for
all TAY
identify as
identify as
identify as
identify as
identify as
_____
_____
_____
_____
_____
1
1
1
1
1
numbers)
(n=XX)
(n=XX)
(n=XX)
(n=XX)
(n=XX)
(n=XX)




Positive Outcomes of

Services
Functioning



1
48
[Insert narrative of YSS-Family Stratified findings, including ranking of findings from best to worst responses,
using data from Table 8]

Table Template 8: Mean Outcomes and Satisfaction scores for all [Organization Name] Caregivers of
TAY participants who completed the YSS-Family, stratified by [gender, race/ethnicity, duration of care].
[Month, Year of YSS-Family Survey].
YSS-Family DOMAINS
Mean scores,
(See Appendix Table 3 for
all TAY
numbers)
(n=XX)


TAY who
TAY who
TAY who
TAY who
TAY who
identify as
identify as
identify as
identify as
identify as
_____
_____
_____
_____
_____
1
1
1
1
1
(n=XX)
(n=XX)
(n=XX)
(n=XX)
(n=XX)

Positive Outcomes of

Services
Functioning



1
49
Findings from the PSS/FSS Supplementary Forms for the MHSIP, YSS-Youth, and YSS-Family

[Insert narrative of MHSIP PSS Supplementary Form findings, including ranking of findings from best to worst
responses, based on data from Table 9]

Table Template 9: PSS Supplementary Form responses for all [Organization Name] TAY participants
who completed the MHSIP [Month, Year of MHSIP Survey].
MHSIP PSS Supplementary form questions
Interacted with a PSS during course of care
Yes
No
PSS Services Received
Provided advice or counseling
Helped me understand what resources were available
Helped me fill out paperwork
Helped me understand what was being asked of me by other staff
Served as a role model
Helped me set goals for recovery
Helped me monitor my progress
Helped me navigate the mental health services system
Provided social support or reduced feelings of isolation
Other
The PSS
Helped me believe I could recover
Understood my experiences
Was easier to speak with than other clinic staff
Was easier to speak with than my doctor
Provided helpful thoughts and insights
Made a difference in my recovery
Had experiences that were similar to my own.

% of TAY participants (n=XX)
Q1
Q1
% of Yes responses (vs. No) from
TAY participants who saw a PSS (n=XX)
(of those who answered yes to Q1)
Q2a
Q2b
Q2c
Q2d
Q2e
Q2f
Q2g
Q2h
Q2i
Q2j
% of Strongly Agree or Agree
responses (vs. Neutral, Disagree, or
Strongly Disagree) from TAY
participants who saw a PSS (n=XX) (of
those who answered yes to Q1)
Q3a
Q3b
Q3c
Q3d
Q3e
Q3f
Q3g

Summary of TAY responses to MHSIP PSS Supplementary Form Open-Ended Questions:
Q2j. What types of help did the Peer Support Specialists/Peer Counselors/Peer Support Partners provide:
Other- please fill in
Q4. What did you like about meeting with a Peer Support Partner?
Q5. What did you dislike about meeting with a Peer Support Partner? Or was there another kind of help you
wanted the Peer Support Partner to provide?
50
[Insert narrative of YSS-Youth PSS Supplementary Form findings, including ranking of findings from best to
worst responses, based on data from Table 10]

Table Template 10: PSS Supplementary Form responses for all [Organization Name] TAY participants
who completed the YSS-Youth, [Month, Year of YSS-Youth Survey].
YSS-Youth PSS Supplementary form questions
Interacted with a PSS during course of care
Yes
No
PSS Services Received
Helped me understand what resources were available for
me
Helped me fill out paperwork
Helped me understand what was being asked of me by
other staff
Served as a role model for me
Helped me set the goals for my treatment
Helped me monitor treatment progress (determine if I was
getting better)
Helped me understand the mental health services system
Provided social support or helped me feel less alone
Helped me get additional services
Other
The PSS
Helped me believe I could get better
Was easier to speak with than my therapist
Provided helpful advice
Made a difference in my treatment

% of TAY participants (n=XX)
Q1
Q1
% of Yes responses (vs. No) from TAY
participants who saw a PSS (n=XX) (of those who
answered yes to Q1)
Q2a
Q2b
Q2c
Q2d
Q2e
Q2f
Q2g
Q2h
Q2i
Q2j
Q2k
% of Strongly Agree or Agree responses (vs.
Neutral, Disagree, or Strongly Disagree)
from TAY participants who saw a PSS (n=XX) (of
Q3a
Q3b
Q3c
Q3d
Q3e
Q3f
Q3g

Summary of TAY responses to YSS-Youth PSS Supplementary Form Open-Ended Questions:
Q2k. What types of help did the PSS/PSP provide: Other- please fill in
Q4. What did you like about meeting with a Peer Support Partner?
Q5. What did you dislike about meeting with a Peer Support Partner? Or was there another kind of help you
wanted the Peer Support Partner to provide?

51
[Insert narrative of YSS-Family FSS Supplementary Form findings, including ranking of findings from best to
worst responses, based on data from Table 11]

Table Template 11: FSS Supplementary Form responses for all [Organization Name] Caregivers of TAY
who completed the YSS-Family, [Month, Year of YSS-Family Survey].
YSS-Family FSS Supplementary form questions
Interacted with a FSS during course of care
Yes
No
FSS Services Received
Helped me understand what resources were available for
my child
Helped me to fill out paperwork
Helped me understand what was being asked of me or
my child by staff
Served as a role model
Helped me monitor my childs treatment progress
Helped me navigate the mental health services system
Provided social support or reduced feelings of isolation
Attended meetings (for example, IEP meetings) with me
Helped me get additional services for my child
Other
The FSS
Helped me believe I could get better
Was easier to speak with than my therapist
Provided helpful advice
Made a difference in my treatment

% of TAY caregivers (n=XX)
Q1
Q1
% of Yes responses (vs. No) from TAY
caregivers who saw a FSS (n=XX) (of those who
answered yes to Q1)
Q2a
Q2b
Q2c
Q2d
Q2e
Q2f
Q2g
Q2h
Q2i
Q2j
Q2k
% of Strongly Agree or Agree responses (vs.
Neutral, Disagree, or Strongly Disagree)
from TAY caregivers who saw a FSS (n=XX) (of
Q3a
Q3b
Q3c
Q3d
Q3e
Q3f
Q3g

Summary of TAY caregivers responses to YSS-Family FSS Supplementary Form Open-Ended
Questions:
Q2k. What types of help did the Family Support Partner/Specialist provide: Other- please fill in
Q4. What did you like about meeting with a Family Support Partner/Specialist?
Q5. What did you dislike about meeting with a Family Support Partner/Specialist? Or was there another kind of
help you wanted the Family Support Partner/Specialist to provide?
52
Discussion: Interpretation of the Findings, Dissemination, and Next Steps

This section is more difficult to provide guidance on, because the authors of this toolkit do not know what
your results will be when you conduct your evaluation. However, we can provide some guidelines and ideas
for how to interpret your findings and turn them into relevant, meaningful recommendations for your
organization.
Process Evaluation Findings- Discussion
In this section, you should summarize your findings by answering the following questions for each of the
Process Evaluation Surveys. Address other questions that emerge from the findings as needed.
Program Manager Survey
- What findings stood out from the Program Manager Survey? (Table Template 1)
- What types of services are commonly provided by PSS/FSS, according to program managers?
Which are less commonly provided?
- What findings from the Program Manager Survey were surprising?
- Are there any clear implications for TAY PSS/FSS services? Think about what you learned about
how PSS/FSS are hired, if the program manager thinks PSS/FSS are cost effective, what types of
trainings the PSS/FSS receive, and what the challenges are for using PSS/FSS in TAY services.
PSS/FSS Survey
- Were there any similarities or differences in the findings of the PSS/FSS Survey compared to the
Program Manager Survey for similar questions? (Table Template 2)
- What were the major strengths and challenges of working as a PSS/FSS from the PSS/FSS
perspective?
- What findings from the PSS/FSS Survey were surprising?
Outcomes Evaluation Findings- Discussion
In this section, you should summarize your findings by answering the following questions for each of the
Outcomes Evaluation Surveys. Address other questions that emerge from the findings as needed. You may
wish to compare your findings to State and National averages for the MHSIP/YSS domains. These data can
be found here: http://www.samhsa.gov/data/sites/default/files/URSTables2014/California.pdf.
MHSIP Outcomes and Satisfaction (TAY in Adult/Older Adult System)
- Most importantly, what proportion of TAY who took the MHSIP had interacted with a PSS/FSS in the
past six months, and did these TAY who interacted with a PSS/FSS report better outcomes than
those who did not? (Table Template 3)
- Did any racial or ethnic groups (or other groups, such as males vs. females) show different patterns
in their outcomes or satisfaction if they had interacted with a PSS or not? (Table Template 6)
YSS-Youth Outcomes and Satisfaction (TAY in Child, Youth, Family System)
- Most importantly, what proportion of TAY who took the YSS-Youth had interacted with a PSS in the
past six months, and did these TAY who interacted with a PSS report better outcomes than those
who did not? (Table Template 4)
- Did any racial or ethnic groups (or other groups, such as males vs. females) show different patterns
in their outcomes or satisfaction if they had interacted with a PSS or not? (Table Template 7)
YSS-Family Outcomes and Satisfaction (Caregivers of TAY in Child, Youth, Family System)
- Most importantly, what proportion of TAY caregivers who took the YSS-Family had interacted with a
FSS in the past six months, and did these TAY caregivers who interacted with a FSS report better
outcomes than those who did not? (Table Template 5)
53
-
-
Did any racial or ethnic groups (or other groups, such as males vs. females) show different patterns
in their outcomes or satisfaction if they had interacted with a FSS or not? (Table Template 8)
Were there any differences in outcomes/satisfaction scores among adult TAY, TAY under age 18,
and TAY caregivers who had interacted with a PSS/FSS? (Table Templates 3, 4, 5, 6, 7, 8)
MHSIP PSS Supplementary Form

- What services were most commonly provided by PSS to TAY who took the MHSIP? (Table
Template 9)
- What aspects of PSS services did TAY who took the MHSIP show the greatest satisfaction with?
- Did TAY who took the MHSIP think that the PSS made a difference in their treatment/recovery?
YSS-Youth PSS Supplementary Form
- What services were most commonly provided by PSS to TAY who took the YSS-Youth? (Table
Template 10)
- What aspects of PSS services did TAY who took the YSS-Youth show the greatest satisfaction with?
- Did TAY who took the YSS-Youth think that the PSS made a difference in their treatment/recovery?
YSS-Family FSS Supplementary Form
- What services were most commonly provided by FSS to TAY caregivers who took the YSS-Family?
- What aspects of FSS services did TAY caregivers who took the YSS-Family show the greatest
satisfaction with? (Table Template 11)
- Did TAY caregivers who took the YSS-Family think that the FSS made a difference in their
treatment/recovery?
- Were there similarities or differences in satisfaction with PSS/FSS services between adult TAY, TAY
under age 18, and TAY caregivers? (Table Templates 9, 10, 11)
Linking the Process and Outcomes Evaluation Findings
In this section, consider what the findings described in the process and outcomes evaluation sections have in
common, or if they contradict one another, describe why you think the findings are similar or different, and
what implication this has for your organizations PSS/FSS programs for TAY.
- What parallel findings did you observe between the Project Manager Survey and the PSS/FSS
survey in the Process Evaluation findings, if any? Do the findings contradict or support each other?
- What parallel findings did you observe between the MHSIP, YSS-Youth, and YSS-Family, and their
corresponding PSS/FSS Supplementary forms in the Outcomes Evaluation findings, if any? Do the
findings contradict or support each other?
- How are the findings between the Process and Outcomes evaluation related, if at all? Can you draw
any parallel conclusions?
- Do any of the findings in the Process and Outcomes evaluation seem to contradict or support each
other? Describe why this might be, and what this means for your organization and TAY services.
- What conclusions can you draw from the evaluation that are supported by multiple aspects of the
Process and/or Outcomes evaluation findings?
Limitations
In this section, you should acknowledge the limitations of your study. This may include the fact that a
convenience sample of TAY was used. This can limit how you interpret the results, as the group of TAY who
did not receive services the week that the survey was implemented may be different from those who did,
such as having less frequent contact with your organization. Another issue might be sample size. If you saw
few statistically significant differences and if the groups you were comparing had fewer than 30 people in
them, low sample size may prevent you from making strong conclusions about the findings. Discuss any
other challenges you faced during data collection or analysis that may affect your ability to interpret the
results, especially as these challenges relate to working with TAY.
54

Next Steps
In this section, you should develop concrete, feasible recommendations for your organization that are clearly
linked to the data. Answering the questions in the three subheadings below will help you accomplish this.

Programmatic/Staff Training Improvements
- If outcomes and satisfaction are better for TAY who interacted with a PSS/FSS, should TAY
PSS/FSS programming be broadened? If there were differences between groups, should the
programs be adapted to meet the needs of a certain subgroup of TAY?
- Will you change your PSS/FSS services for TAY and/or TAYs families? In what ways? Are these
anticipated changes feasible, and will they be accepted by staff and TAY consumers?
- Will you seek additional training for PSS/FSS or other employment-related changes?

Further Research and Evaluation Efforts
-
Will these findings lead to future quality improvement activities, or is additional evaluation needed?
What areas of interest did this evaluation reveal that need more study?

Disseminating the Findings
-
Will you share these results with a broader audience? In what medium/venue? With whom? What is
your expectation for this dissemination- advocacy? Funding? Options for dissemination might include
presenting the results at a state or national conference about mental health, TAY, evaluation, or
health services. The TAY Evaluation and Quality Improvement Summit may be a venue in which
other organizations that have used the toolkit can meet and share their results. The results may also
be used in grant applications to secure funding for programming, program development, or further
evaluation and research into peer support and TAY. Finally, it is important to share the findings with
your program managers, PSS/FSS, and TAY consumers. Make a short factsheet that can be posted
at your program sites, or have a meeting with program managers and PSS/FSS to discuss the
results with them. Then, encourage the PSS/FSS to share these results with the consumers they
work with, if they feel it is appropriate.

55
TOOLKIT SUPPORT AND FEEDBACK

Frequently Asked Questions (FAQ)
Can I use this toolkit for more than just our organizations TAY population?
Yes, this evaluation can be adapted to focus on more broadly defined or different populations, such as the
entire consumer population, children and families, adults, older adults, different racial/ethnic groups, or for
individuals with specific diagnoses. This toolkit is based on an evaluation that was implemented for all
consumers in San Diego Countys Behavioral Health Services.

For the Process Evaluation, users will also have to adapt the Program Manager and PSS/FSS surveys to
address PSS/FSS in general, and not just those that work with TAY and TAY caregivers. For the Outcomes
Evaluation, the supplementary PSS/FSS pages to the YSS or MHSIP can be implemented with all
consumers/caregivers instead of just with TAY and TAY caregivers, and stratified analyses can be
conducted using the data to present findings for different age groups (e.g., children and their caregivers, TAY
and their caregivers, adults, older adults). Adjust the evaluation according to your organizations needs and
how PSS/FSS are used by local providers and programs.

We do not have any or few family support specialists (FSS) that work with TAY caregivers, do we
need to implement the FSS portions of the evaluation?
The FSS portions of the evaluation were included since some TAY who remain in the child/youth/family
system of care may have caregivers who are involved in treatment. However, this is not common. Toolkit
users may exclude the FSS portion of the evaluation if there are few or no FSS who work with TAY
caregivers or to reduce the overall burden of the evaluation.

We are not sure if we can get access to the MHSIP data in a timely fashion to conduct this evaluation-
what are our options?
Section 2e (Data Entry and Management) includes a description of how others have obtained MHSIP/YSS
data by entering it themselves before sending the surveys to the state (page 11). If these options are not
feasible for your organization at this point, you may wish to conduct only the process evaluation portion of
this toolkit. Another option would be to use a different measure of outcomes and satisfaction paired with the
MHSIP/YSS supplemental pages.
Do you have feedback or more questions? Contact the Toolkit Authors

Do you have feedback or questions about this evaluation toolkit? Specifically, we would like to know:
- Have you implemented the evaluation described in this toolkit, and if so, what was your experience?
o Did you identify any best practices in the implementation of your evaluation? Were there
any unforeseen challenges?
o How did TAY and other stakeholders react to the evaluation, such as during data collection
or disseminating the findings?
o Did the evaluation result in any important changes to your programs?
o How can we make the toolkit or the evaluation described in the toolkit better?
- If you did not implement the evaluation described in this toolkit, why not?
- Were any parts of the toolkit (e.g., step-by-step guide, feasibility checklist, external evaluation
resources lists, MS Excel data entry template) particularly useful? Any parts that were not helpful?
- Is there information that would be helpful addition to the toolkit?

Please direct your feedback or inquiries to the toolkits authors via email at teqisummit@gmail.com or by
filling out the feedback form on the TEQI Summit website at http://www.teqisummit.org/contact.html.

56
EXTERNAL EVALUATION RESOURCES

Designing and Implementing Evaluations

The table below includes titles and descriptions of external resources, such as manuals, guides, and
webinars to help understand how to plan and implement evaluations.

Resource
Description
Link
W.K. Kellogg Foundation Evaluation
Handbook
Introduction to Program Evaluation for
Public Health Programs: A Self-Study
Guide

Contra Costa County Health Services
Mental Health Divisions Program
Evaluation Framework
Programs and Tools to Improve the
Quality of Mental Health Services,
Agency for Healthcare Research and
Quality (AHRQ), Research in Action
Mental Health Factsheet #16
Comprehensive guide to conducting health

and social program evaluations from a non-
profit perspective.

Comprehensive guide to conducting Public
Health program evaluations.
https://www.wkkf.org/resource-
directory/resource/2010/w-k-kellogg-
foundation-evaluation-handbook
Guidelines for conducting evaluations of

mental health programs from the perspective
of a California county.
http://cchealth.org/mentalhealth/pdf/2010_062
9_program_evaluation_framework.pdf
Information about evaluation and quality

improvement tools developed through AHRQ-
funded research for Mental Health programs.
http://archive.ahrq.gov/research/findings/facts
heets/mental/issue16/issue16.pdf
http://www.cdc.gov/eval/guide/

Data Collector and Data Entry/Management Training Resources
The following table includes resources on how to train data collectors, how to enter data, and how to
manage your data effectively to make data analysis easier.

Resource
Description
Link
Human Subjects Research Ethics Field
Training Guide
Merritt, M., Labrique, A.B., Katz, J., Rashid,
M., West, K.P., Pettit, J. (2010). Johns
Hopkins Bloomberg School of Public Health
This brief guide provides training

information for individuals who will collect
data from human subjects. Although
evaluations are not considered research,
this type of data collector training can
promote a greater degree of
methodological rigor and ethical
treatment of subjects for evaluation and
quality improvement projects that involve
direct contact with consumers.
Preparing Data in Excel
This website provides guidelines and
Center for Collaboration on Research Design best practices on how to set up a
and Analysis, College of Public Health,
usable database in Excel.
University of Nebraska Medical Center

Excel and Questionnaires: How to enter
This YouTube video tutorial shows the
the data and create the charts
user how to take a completed survey and
Macnamara, D. (2013). YouTube.
enter the data into an Excel worksheet.

Excel for Evaluators Tutorials
This series of YouTube tutorial playlists
Importing and Merging Datasets
(i.e., groups of related videos) shows
Cleaning Data and Recoding Variables
users how to import/merge datasets,
Housekeeping Skills in Excel (aka Getting clean/recode variables, and organize
Your Spreadsheet Organized and Ready spreadsheets in Excel to make data
to Use)
analysis easier.
Emery, A.K. (2014). YouTube.
English version available at:

http://www.jhsph.edu/offices-and-
services/institutional-review-board/_pdfs-and-
docs/field-guide-2010-02-25.pdf

Other language versions available at:
http://www.jhsph.edu/offices-and-
services/institutional-review-board/training/field-
training-guides-for-data-collectors/

Available at:
http://www.unmc.edu/publichealth/centers/ccorda/e
xceldata.html
Available at:
https://www.youtube.com/watch?v=uVGD_5Tk6ao
Importing and Merging Datasets Playlist
https://www.youtube.com/playlist?list=PLZzQj_sVf3
skzJvklgHr4qguKRGr2lbFb

Cleaning Data and Recoding Variables Playlist
smyMAaEixeEegG9b8orRyeW

Housekeeping Skills in Excel (aka Getting Your
Spreadsheet Organized and Ready to Use)
Playlist
sl5zTBpadKjIICiw5yHUGwN
57
Data Analysis Resources

The table below lists guides and tutorials that help the user understand how to analyze their data, with
an emphasis on using Excel, a software that is widely available and which many individuals already
know how to use.
Resource
Description
Link
Using Excel for Statistical Data
Analysis Caveats
Goldwater, E. (2007), Biostatistics
Consulting Center, University of
Massachusetts School of Public
Health.
This website provides guidance on using

Excel for statistical analysis using the
Data Analysis Toolpak. This includes
descriptive statistics, correlations,
statistical significance tests (e.g., t-tests),
linear regressions and analysis of
variance (ANOVA).

Using Excel for Statistical Analysis This short tutorial document uses
Doig, S. (2013). Presentation at 2013 screenshots to show users how to
Investigative Reporters and Editors
conduct statistical analyses in Excel
Conference.
using the Data Analysis Toolpak.,

including statistical significance tests and
regression analyses.
Available at: http://people.umass.edu/evagold/excel.html
Excel Statistical Analysis Tutorials:

- One and Two-Way ANOVA
- T-tests of dependent and
independent groups
- Chi Square tests
- Linear Regression
Longstreet, D. Youtube user name:
statisticsfun

These YouTube tutorial playlists clearly

and concisely illustrate how different
statistical tests are conducted in Excel.
Each playlist includes information on
each type of statistical analysis, and
instructions on how to conduct each
statistical analysis in Excel. Some
playlists include additional videos on how
to conduct the analysis in SPSS.
Excel for Evaluators Tutorials

- Descriptive Statistics in Excel
- Putting It All Together
(Intermediate & Advanced Excel
Techniques)
- Charts in Excel
Emery, A.K. (2014). YouTube.
These YouTube tutorial playlists cover

generating basic descriptive statistics,
analyzing satisfaction survey data, and
creating charts in Excel.
One-way ANOVA Playlist

https://www.youtube.com/playlist?list=PL3A0F3CC5D484
31B3

Two-way ANOVA (Factorial Analysis) Playlist
https://www.youtube.com/playlist?list=PLWtoq-
EhUJe2TjJYfZUQtuq7a0dQCnOWp

T-tests of independent and dependent groups Playlist
(4 videos)
https://www.youtube.com/playlist?list=PL8B759A5C1C5C
12AF

Chi Square Test Playlist
https://www.youtube.com/playlist?list=PLD2EE7A628436
4CA2

Linear Regression Playlist
https://www.youtube.com/playlist?list=PLF596A4043DBE
AE9C

Descriptive Statistics in Excel Playlist
https://www.youtube.com/playlist?list=PLZzQj_sVf3slETv
ed-MKURPhodMP9DNu7

Putting It All Together (Intermediate & Advanced
Excel Techniques) Playlist
https://www.youtube.com/playlist?list=PLZzQj_sVf3snaMd
kgN_Qh1DaYTMGhs8OV

Charts in Excel Playlist
https://www.youtube.com/playlist?list=PLZzQj_sVf3snGd
Cq94HKyMsz6OzFS2aDg

Available at:
http://ire.org/media/uploads/car2013_tipsheets/excel_stat
s_nicar2013.pdf
58
APPENDIX
Starting on next page

59
Outcomes Evaluation Measure Domains

Table 1: Adult MHSIP Domain Items
23
Adult MHSIP Domains

General Satisfaction:
I liked the services that I received here.
If I had other choices, I would still get services at this agency.
I would recommend this agency to a friend or family member.
Perception of Access:
The location of services was convenient.
Staff were willing to see me as often as I felt it was necessary.
Staff returned my calls within 24 hours.
Services were available at times that were good for me.
I was able to get all the services I thought I needed.
I was able to see a psychiatrist when I wanted to.
Perception of Quality and Appropriateness:
Staff believed that I could grow, change and recover.
I felt free to complain.
I was give information about my rights.
Staff encouraged me to take responsibility for how I live my life.
Staff told me what side effects to watch for.
Staff respected my wishes about who is and is not to be given information
about my treatment.
Staff were sensitive to my cultural/ethnic background.
Staff helped me obtain the information needed so I could take charge of
managing my illness.
I was encouraged to use consumer-run programs.
Perception of Outcomes:
I deal more effectively with daily problems.
I am better able to control my life.
I am better able to deal with crisis.
I am getting along better with my family.
I do better in social situations.
I do better in school and/or work.
My housing situation has improved.
My symptoms are not bothering me as much.
Functioning:
My symptoms are not bothering me as much.
I do things that are more meaningful to me.
I am better able to take care of my needs.
I am better able to handle things when they go wrong.
I am better able to do things that I want to do.
Social Connectedness:
I am happy with the friendships I have.
I have people with whom I can do enjoyable things.
I feel I belong in my community.
In a crisis, I would have the support I need from family or friends.
Perception of Participation in Treatment Planning:
I felt comfortable asking questions about my treatment/medications.
I, not staff, decided my treatment goals.
Arrests/Police Interactions:
Since you began receiving services, have your encounters with the police:
Demographics
What is your gender?
Are you of Mexican/Hispanic/ Latino origin?
What is your race?
What is your date of birth?
Approximately, how long have you received services here? (Duration of
Services)
Question Number,
2015 MHSIP
Survey

Page 1, Q1
Page 1, Q2
Page 1, Q3

Page 1, Q4
Page 1, Q5
Page 1, Q6
Page 1, Q7
Page 1, Q8
Page 1, Q9

Page 1, Q10
Page 1, Q12
Page 1, Q13
Page 1, Q14
Page 1, Q15
Page 1, Q16
Response Options

Strongly Agree; Agree; I am Neutral;
Disagree; Strongly Disagree; N/A


Page 1, Q18
Page 1, Q19
Page 1, Q20

Page 1, Q21
Page 1, Q22
Page 2, Q23
Page 2, Q24
Page 2, Q25
Page 2, Q26
Page 2, Q27
Page 2, Q28

Page 2, Q28
Page 2, Q29
Page 2, Q30
Page 2, Q31
Page 2, Q32

Page 2, Q33
Page 2, Q34
Page 2, Q35
Page 2, Q36

Page 1, Q11
Page 1, Q17

Page 4, Q4 and Q7

Page 4, Q8
Page 4, Q9
Page 4, Q10
Page 5, Q11
Page 4, Q1





Reduced; Stayed the same; Increased; N/A

Female; Male; Other
Yes; No; Unknown
American Indian/AK Native; Asian; Black/
African American; Native HI/Other Pacific
Islander; White/Caucasian; Other; Unknown
MM-DD-YYYY
1st visit; >1 visit but received services for <1
mo.; 1-2 mo.; 3-5 mo.; 6 mo. to 1 yr.; >1 yr.
23
Allard, L. (2014). MHSIP Survey Analysis by Planning and Policy Region: An evaluation of parent/guardian satisfaction with
community mental health services. Tennessee Department of Mental Health and Substance Abuse Services, Office of Research.
Available at: www.tamoc.org.
60
Table 2: YSS-Youth Domain Items

24
YSS-Youth Domains
Satisfaction with Services:
Overall, I am satisfied with the services I received.
The people helping me stuck with me no matter what.
I felt I had someone to talk to when I was troubled.
I received services that were right for me.
I got the help I wanted.
I got as much help as I needed.
Participation in Treatment:
I helped to choose my services.
I helped to choose my treatment goals.
I participated in my own treatment.
Good Access to Service:
The location of services was convenient for me.
Services were available at times that were good for me.
Cultural Sensitivity:
Staff treated me with respect.
Staff respected my religious/spiritual beliefs.
Staff spoke with me in a way that I understood.
Positive Outcomes of Services:
I am better at handling daily life.
I get along better with family members.
I get along better with friends and other people.
I am doing better in school and/or work.
I am better able to cope when things go wrong.
I am satisfied with my family life right now.
Functioning:
I am better at handling daily life.
I get along better with family members.
I get along better with friends and other people.
I am doing better in school and/or work.
I am better able to cope when things go wrong.
I am better able to do things I want to do.
I know people who listen and understand me when I need to talk.
I have people that I am comfortable talking with about my problems.
Since you began receiving mental health services, have your
encounters with the police
Demographics
What is your gender?
Are you of Mexican/Hispanic/ Latino origin?
What is your race?
What is your date of birth?

Approximately, how long have you received services here? (Duration
of Services)
Question Number, 2015

YSS-Youth Survey

Page 1, Q1
Page 1, Q4
Page 1, Q5
Page 1, Q7
Page 1, Q10
Page 1, Q11

Page 1, Q2
Page 1, Q3
Page 1, Q6

Page 1, Q8
Page 1, Q9

Page 1, Q12
Page 1, Q13
Page 1, Q14
Page 1, Q15

Page 1, Q16
Page 1, Q17
Page 1, Q18
Page 1, Q19
Page 1, Q20
Page 1, Q21

Page 1, Q16
Page 1, Q17
Page 1, Q18
Page 1, Q19
Page 1, Q20
Page 1, Q22

Page 2, Q23
Page 2, Q24
Page 2, Q25
Page 2, Q26

Page 3, Q7 and Q13

Page 4, Q17
Page 4, Q18
Page 4, Q19
Page 4, Q20
Page 2, Q4
Response Options

Strongly Agree; Agree; Undecided;







Been reduced; Stayed the same;
Increased; N/A

Female; Male; Other
Yes; No; Unknown
American Indian/AK Native; Asian;
Black/ African American; Native
HI/Other Pacific Islander;
White/Caucasian; Other; Unknown
MM-DD-YYYY
st
1 visit; >1 visit but received
services for <1 mo.; 1-2 mo.; 3-5
mo.; 6 mo. to 1 yr.; >1 yr.
24
Ibid.
61
Table 3: YSS-Family Domain Items

25
YSS-Family Domains
Satisfaction with Services:
Overall, I am satisfied with the services my child received.
The people helping my child stuck with us no matter what.
I felt my child had someone to talk to when he/she was troubled.
The services my child and/or family received were right for us.
My family got the help we wanted for my child.
My family got as much help as we needed for my child.
Participation in Treatment:
I helped to choose my childs services.
I helped to choose my childs treatment goals.
I participated in my childs treatment.
Good Access to Service:
The location of services was convenient for us.
Services were available at times that were good for us.
Cultural Sensitivity:
Staff treated me with respect.
Staff respected my family's religious/spiritual beliefs.
Staff spoke with me in a way that I understood.
Positive Outcomes of Services:
My child is better at handling daily life.
My child gets along better with family members.
My child gets along better with friends and other people.
My child is doing better in school and/or work.
My child is better able to cope when things go wrong.
I am satisfied with our family life right now.
Functioning:
My child is better at handling daily life.
My child gets along better with family members.
My child gets along better with friends and other people.
My child is doing better in school and/or work.
My child is better able to cope when things go wrong.
My child is better able to do things he or she wants to do.
I know people who listen and understand me when I need to talk.
I have people that I am comfortable talking with about my child's problems.
Since your child began receiving mental health services, have their
encounters with the police
Demographics
What is your childs gender?
Are either of the childs parents of Mexican/Hispanic/ Latino origin?
What is your childs race?
What is your childs date of birth?

Approximately, how long has your child received services here? (Duration of
Services)
Question Number,
2015 YSS-Family
Survey

Page 1, Q1
Page 1, Q4
Page 1, Q5
Page 1, Q7
Page 1, Q10
Page 1, Q11

Page 1, Q2
Page 1, Q3
Page 1, Q6

Page 1, Q8
Page 1, Q9

Page 1, Q12
Page 1, Q13
Page 1, Q14
Page 1, Q15

Page 1, Q16
Page 1, Q17
Page 1, Q18
Page 1, Q19
Page 1, Q20
Page 1, Q21

Page 1, Q16
Page 1, Q17
Page 1, Q18
Page 1, Q19
Page 1, Q20
Page 1, Q22

Page 2, Q23
Page 2, Q24
Page 2, Q25
Page 2, Q26

Page 3, Q8 and Q14

Page 4, Q18
Page 4, Q19
Page 4, Q20
Page 4, Q21
Page 2, Q5
Response Options








Been reduced; Stayed the same;
Increased; N/A

Female; Male; Other
Yes; No; Unknown
American Indian/AK Native; Asian;
Black/ African American; Native
HI/Other Pacific Islander;
White/Caucasian; Other; Unknown
MM-DD-YYYY
st
1 visit; >1 visit but received
services for <1 mo.; 1-2 mo.; 3-5
mo.; 6 mo. to 1 yr.; >1 yr.
25
Ibid.
62
Evaluation Measures

63
Program Manager Peer and Family Support Specialist Survey

Exploring Peer and Family Support Services - Program Manager Feedback

Clinics in Behavioral Health Services systems often have former or current consumers performing the role of
Peer Support Specialist (PSS) and supportive family members in the role of Family Support Specialist
(FSS).

A Peer Support Specialist is..."Someone who has progressed in their own recovery from mental health or
behavioral health challenges and can now offer professional services to mental or behavioral health
consumers. Because of their life experiences, a PSS provides expertise that professional training cannot
replace." NOTE: In the children's system of care, these specialists are often called Peer Support Partners.

A Family Support Specialist is... "Someone who has personal experiences as a caregiver to a family
member with mental or behavioral health challenges. They use this experience to provide hope and
education to other people who have family members with mental or behavioral health challenges and
encourage them to support their loved ones." NOTE: In the children's system of care, these specialists are
often called Family Support Partners.

PSSs and FSSs help bridge the gap between an individual's needs and programs' ability to meet those
needs. PSSs and FSSs offer support to persons experiencing mental health challenges and/or their family
members from the unique perspective of "someone who's been there." They provide a resource to programs
and clinics that can potentially expand the services and insights available to mental health and behavioral
health consumers and their families.

Given their widespread presence in the behavioral health system in many counties throughout the U.S., it is
important to assess the presence, function, and effectiveness of these specialists.

Please answer the following questions to help us better understand the influence of PSSs and FSSs
within the Behavioral Health Services system.

NOTE: Please count all PSSs and FSSs in your responses (including those PSSs and FSSs who work on
your site, but may be employed by another agency). For some questions, you will be asked to think only of
the PSS/FSS who work with TAY and/or their families.

64
1. Which Behavioral Health Services system of care do you work with? Select one answer.
o Children, Youth, and Families Behavioral Health Services (CYFBHS)
o Adult and Older Adult Behavioral Health Services (AOABHS)
o Both CYFBHS and AOABHS

2. Your job title/role? _____________________________

3. Program Name? Provide the FULL NAME of your program. No abbreviations please.

____________________________________________________

4. Types of services your program provides? * Check all that apply.
Assertive Community Treatment (ACT)
Behavioral Health (BH) Court
Child and Adolescent Psychiatric Services (CAPS)
Case Management
Case Management - Institutional
Case Management - Strengths
Case Management - Transitional
Day Treatment
Fee for Service (FFS)
Juvenile Forensic Services (JFS)
Outpatient
Prevention (PEI)
Emergency Psychiatric Unit (EPU)
Crisis Residential
Residential
Therapeutic Behavioral Services (TBS)
Inpatient Hospital
Other: ___________________________

5. Do you have Peer Support Specialists (PSSs) and/or Family Support Specialists (FSSs) at
your facility? Please respond 'yes' if you have any staff where "lived experience' is a requirement
for the job with TAY. NOTE: Your program may have a different job title. If so, please answer item 6.
Select one answer.
o Yes, both PSSs and FSSs.
o Yes, only PSSs.
o Yes, only FSSs.
o No (If your answer is 'No', mark 'No' to the left and then skip to the end of the survey).

6. If your program does not use the job title "Peer Support Specialist" and "Family Support
Specialist," what job title is used for the staff members that are required to have "lived
experience"? ____________________________

65

7. Please indicate the total number of PSSs and FSSs in your program. _____________

8. Do you utilize Peer Support Specialists (PSSs) and/or Family Support Specialists (FSSs) with
transition age youth (TAY, ages 16-25) at your facility? Select one answer. Please respond 'yes'
if you have any staff where "lived experience' is a requirement for the job with TAY. NOTE: Your
program may have a different job title. If so, please answer item 10.
o Yes, both PSSs and FSSs.
o Yes, only PSSs.
o Yes, only FSSs.
o No (If your answer is 'No', click 'No' to the left and then skip to the end of the survey).

9. If your program does not use the job title "Peer Support Specialist" and "Family Support
Specialist," what job title is used for the staff members who work with TAY that are required
to have "lived experience"? ____________________________

10. Please indicate the total number of PSSs and FSSs in your program who work with TAY.
_____________

11. Which functions do the PSS/FSS at your facility perform in providing services to TAY and/or
their families? Check all that apply.
Provide advice or counseling to consumers.
Help consumers understand what resources are available.
Help consumers fill out paperwork.
Help consumers understand what staff is asking of them.
Being a role model (for recovery).
Help to create/set recovery goals.
Help with monitoring progress.
Help with navigating the behavioral health services system.
Provide social and/or emotional support.
Administrative/clerical.
Help coordinating physician visits and other medical appointments.
Arranging transportation to and from medical services.
Providing education about medical conditions and recovery strategies.
Facilitating communication with health care providers.
Maintaining telephone contact between patients and health care providers.
Motivate and educate consumers about the importance of preventive services.
Providing education to improve health literacy.
Assisting with the financing and management of medication.
Other: ______________________________

66

12. Are most of the PSSs or FSSs who work with TAY at your facility volunteers or do they get
paid? Select one answer.
o Paid (click, then continue on with 21 and 22)
o Non-paid volunteers (click, then skip to question 17)

13. If your PSSs/FSSs who work with TAY are paid, do you feel that it is cost effective for your
program? Select one answer.
o Yes
o No

14. Do your paid PSS/FSS work full time, part time, both?
o All PSS/FSS are full time
o All PSS/FSS are part time
o Some PSS/FSS are full time, some are part-time

15. If your program had the resources, would you increase the wages of the PSSs and FSSs who
work with TAY at your facility? Select one answer.
o Yes
o No

16. Are the PSSs/FSSs who work with TAY on your site contracted by another agency, paid out of
your program's contract budget, or both? Check all that apply.
Contracted by another agency.
Paid out of our program's contract budget.

17. What are your hiring plans for PSSs/FSSs for TAY and/or TAYs families in the next 12
months? Select one answer.
o Downsize by several (3+)
o Downsize a little (1-2)
o Stay with what we've got now
o Increase a little (1-2)
o Increase by several (3+)
o Unsure
o Other: _________________________

67
18. What kind of training do your PSSs who work with TAY typically have? Check all that apply.
Other: _________________________

19. What kind of training do your FSSs who work with TAY and/or their families typically have?
Check all that apply.
Other trainings through BHETA (Behavioral Health and Training Academy)
Other: ________________________

20. Are your PSSs who work with TAY certified? Select one answer.
o Yes, all are certified.
o Yes, some are certified.
o No, none are certified. (click, then skip to question 29)
o Don't know or not sure.

21. What, if any, additional trainings would you like to see offered to PSSs and FSSs who work
with TAY and/or TAYs families? Please describe in detail.
________________________________________________________________________________
________________________________________________________________________________
________________________________________________________________________________
________________________________________

22. From where do you typically recruit or hire your PSSs and FSSs who work with TAY and/or
TAYs families? Please describe in detail.
________________________________________________________________________________
________________________________________________________________________________
________________________________________________________________________________
________________________________________

68

23. What is an average consumer caseload for a PSS who work with TAY in one week? Please
type a number 0-99 in the box below.
____________

24. What is an average consumer case load for an FSS who work with TAY and/or TAYs families
in one week? Please type a number 0-99 in the box below.
____________

25. On average, how long does a TAY consumer get to spend with a PSS per session? Select all
that apply.
0-15 minutes
16-30 minutes
31-45 minutes
46-60 minutes
61-90 minutes
90+ minutes

26. On average, how long does a TAY consumer /TAYs family member get to spend with an FSS
per session? Select all that apply.
0-15 minutes
16-30 minutes
31-45 minutes
46-60 minutes
61-90 minutes
90+ minutes

27. How do the PSSs/FSSs who work with TAY and/or TAYs families at your facility typically
work? Check all that apply.
One-on-one
In groups as a lead or co-facilitator
In groups as a support to someone else who is facilitating
Other: ________________

28. How would you rate your experience finding and hiring PSSs/FSSs staff who work with TAY
and/or TAYs families? Select one answer.
o Very difficult
o Somewhat difficult
o Neutral
o Somewhat easy
o Very easy

69
29. How would you rate your experience in being able to retain PSSs/FSSs staff who work with
TAY and/or TAYs families in your program? Select one answer.
o Very difficult
o Neutral
o Somewhat easy
o Very easy

30. What QUALIFIES or DISQUALIFIES a TAY consumer/caregiver as someone who gets
assistance from a PSS/FSS? Explain below.
________________________________________________________________________________
________________________________________________________________________________
________________________________________________________________________________
________________________________________

31. Are there any drawbacks or limitations in having PSSs and/or FSSs who work with TAY
and/or TAYs families at your program? Explain below.
________________________________________________________________________________
________________________________________________________________________________
________________________________________________________________________________
________________________________________

32. Have any of the PSSs or FSSs who work with TAY and/or TAYs families advanced to a
different job within your program? Select one answer.
o Yes
o No

33. What advancement possibilities do you see for PSSs/FSSs who work with TAY and/or TAYs
families in your program? Please also describe the limitations and/or barriers related to
advancement possibilities, such as trainings/certifications needed.
________________________________________________________________________________
________________________________________________________________________________
________________________________________________________________________________
________________________________________

34. Do you have any additional comments that you'd like to share with us about your or your
program's experience with PSSs or FSSs who work with TAY and/or TAYs families?
________________________________________________________________________________
________________________________________________________________________________
________________________________________________________________________________
________________________________________

Thank you for helping us develop a better picture of Peer and Family Support Services for TAY. We
appreciate your feedback and your time. Have a great day! For any questions regarding this survey,
please feel free to contact us at: [evaluationmanageremail@organizationname.org]

70
Peer/Family Support Specialist Survey

In the Adult and Older Adult System of Care, a Peer Support Specialist (PSS) is someone who has
progressed in their own recovery from mental or behavioral health challenges and can now offer professional
services to other mental or behavioral health consumers. A Family Support Specialist (FSS) is someone
who has personal experience as a caregiver to a family member with mental health or behavioral health
challenges. Family Support Specialists (FSSs) use this experience to provide hope and education to other
people who have family members with mental health or behavioral health challenges and encourage them to
support their loved ones.

In the Children, Youth, and Families System of Care, a Peer Support Partner / Peer Support Specialist
(PSP/PSS) is someone who has received mental health services before and is using their lived experience to
help others. A Family Support Specialist (FSS) or Family Support Partner (FSP) is either a caregiver of a
child/youth who is a consumer in a public agency serving children or an individual with experience as a
consumer in a public agency serving children.

Because of their life experiences, Peer and Family Support Specialists/Partners provide expertise that
professional training cannot replace.

Given the widespread presence of Peer and Family Support Specialists/Partners in the Behavioral Health
System in many counties throughout the U.S., it is important to get feedback from the Peers themselves.

If you are employed as a Peer and Family Support Specialist/Partner within the Behavioral Health
Services system, please answer the questions below to help us better understand your experiences.

* Required

71
1. Which Behavioral Health Services system do you work with? * Select one answer.
o Children, Youth, and Families Behavioral Health Services (CYFBHS)
o Adult and Older Adult Behavioral Health Services (AOABHS)
o Both CYFBHS and AOABHS

2. Please choose the category that best describes you: * Select all that apply.
I am a current or former consumer of mental health services (or I am an individual with lived
experience with mental health challenges)
I am the parent or caregiver of a current or former consumer of mental health services
If checked:
Parent/caregiver of youth (17 and under)
Parent/caregiver of adult (18 and older)
I am a non-parental family member of a current or former consumer of mental health services
(e.g., spouse, sibling, etc.)
If checked:
Non-parental family member of youth (17 and under)
Non-parental family member of adult (18 and older)
Other (e.g., friend, neighbor, etc.): ___________

**The following questions ask about your current position. If you currently hold multiple positions as
an employee or volunteer, please provide responses for the role in which you spend the most time. If
you spend equal time between your positions, respond for the role which is most relevant to your
career.**

3. Please choose the category that best describes the consumers served where you are
employed and/or are volunteering: Select all that apply.
Transition Age Youth (TAY) only (services targeted specifically towards those in the TAY age
ranges of 16 through 25) and NOT their families/caregivers (continue with survey)
TAY and/or families/caregivers of TAY (services targeted specifically towards those in the TAY
age ranges of 16 through 25) (continue with survey)
Children (ages 0-5) and/or their families/caregivers (skip to end of survey)
Older children and adolescents (ages 6-15) and/or their families/caregivers (skip to end of
survey)
Adult and Older Adult (ages 25+) (skip to end of survey)
Older Adult specific services (ages 60+) (skip to end of survey)

If you work with TAY age 16-25 and/or their families, please continue the survey. When answering,
focus on your work with TAY and their families.

If you do not work with TAY age 16-25 and/or their families, you may end the survey now.
72
SECTION A: TAY Services

Please read each of the services listed below and consider the following questions: A. How often do
the TAY consumers and/or their families/caregivers you work with need these types of services? B.
How often do you typically provide these types of services in your role as Peer/Family Support
Specialist for TAY? Keep in mind that this is not an evaluation of your work as a Peer/Family Support
Specialist; sometimes there are outside factors that prevent a needed service from being provided,
such as lacking funds to get certification to provide a service.

4. How often do the TAY consumers or families you work with NEED each of these types of
services?
Never Sometimes Often Always

a. Coordinating physician visits and/or other
O
O
O
O
mental or physical health appointments.

b. Arranging transportation to and from mental or
O
O
O
O
physical health services.

c. Accessing and maintaining insurance coverage.
O
O
O
O

d. Providing education about mental health
O
O
O
O
problems and recovery/management strategies.

e. Facilitating communication with mental health
O
O
O
O
care providers.

f. Maintaining telephone contact between
O
O
O
O
consumers and mental health care providers.

g. Motivating and educating individuals or their
O
O
O
O
family/caregivers about the importance of
preventive services.

h. Assisting individuals/families/caregivers in
O
O
O
O
completing medical, financial, and other forms.

i. Coordinating care among providers.
O
O
O
O

j. Arranging for translation services.
O
O
O
O

k. Providing education to improve mental health
O
O
O
O
literacy. (Help understanding basic health
information so that someone can make decisions
about their health.)

l. Providing emotional support.
O
O
O
O

m. Assisting with medication management and
O
O
O
O
financing.

n. Assisting with issues related to housing.
O
O
O
O

o. Assisting with issues related to employment.
O
O
O
O

73
5. Now, how often do you typically PROVIDE this type of service in your role as a Peer/Family
Support Specialist?

O
O
O
O

O
O
O
O

O
O
O
O

O
O
O
O

O
O
O
O
care providers.

O
O
O
O

O
O
O
O

O
O
O
O

O
O
O
O

O
O
O
O

O
O
O
O

O
O
O
O

O
O
O
O
financing.

O
O
O
O

O
O
O
O


O
O
O
O

O
O
O
O

O
O
O
O

74


care providers.








financing.



75
SECTION B
6. What is it like to work as a Peer/Family Support Specialist for TAY and/or their families?
Please enter one response per row:

Strongly Somewhat Somewhat Strongly
agree
agree
disagree disagree
a. I have a clear job description.
b. I am clear about what I can and cannot

do in my role as a Peer/Family Support
Specialist.
c. Identifying as both a consumer and a
staff member is challenging for me.
d. I identify with the consumers more

than with other staff.
e. I receive high quality supervision.
f. I receive enough supervision.

g. I receive the individual support I need.
O
O
O
O
O
O
O
O
h. I am afraid to ask for help.
i. I feel comfortable discussing my

diagnosis with others.
j. I experience burnout.
k. I experience feelings of isolation in my

role as Peer/Family Support Specialist.
l. I get paid an adequate amount for the

services I provide.
m. I experience benefits from interacting

with consumers.
n. I am recognized as a valuable member

of the team by the non-Peer/Family
Support Specialist staff.
o. I feel stigmatized by the non-

p. I think my presence here benefits the

other staff.
q. I think I am a positive role model of a

consumer in recovery for the non-
r. It seems like the non-Peer/Family
Support Specialist staff do not like
mental health consumers.
s. I have good communication with other
staff.
t. I feel like a colleague with the other
staff.
u. The culture where I work is Peer/Family
Support Specialist friendly.
76
SECTION C: Additional Feedback

7. Which, if any, of these trainings have you completed? Check all that apply.

Medication for success
Other: _____________

8. How would you rate your experience finding a job as a Peer/Family Support Specialist?
Select one answer.
o Very difficult
o Neutral
o Somewhat easy
o Very easy

9. When you meet with for TAY and/or their families, where are your sessions most typically
held? Select one answer.
o In a designated private office or room
o In any available private office or room
o In the waiting room of the clinic or hospital
o At a designated desk in a larger shared common room
o In a shared common room
o In the hallway or corridor
o Other: ________________

10. How much money do you make per hour working as a Peer/Family Support Specialist for
TAY and/or their families?
(Round to the nearest dollar.)
$____________

77
DRAFT- NOT FOR DISTRIBUTION OR CITATION

11. Typically, how many hours per week do you work as a Peer/Family Support Specialist for TAY
and/or their families? Select one answer.
o Less than 5 hours per week
o 5-10 hours per week
o More than 40 hours per week

12. If your program could offer you more hours, would you want them? Select one answer.
o Yes
o No

13. Would you be concerned about losing your benefits if you worked more hours or made more
money? Select one answer.
o Yes
o No
o N/A

14. Are you interested in advancing your career to another type of job within the County of San
Diego Behavioral Health Services? Select one answer.
o Yes
o No

15. If so, please list the job title and please describe the training you feel would be appropriate to get
you there.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
______________________________________________________

16. Has your perception of being a Peer/Family Support Specialist improved, worsened, or remained
the same since you started working as a Peer/Family Support Specialist for TAY and/or their
families? Select one answer.
o Improved
o Worsened
o Remained the same

78
17. What do you LIKE about being a Peer/Family Support Specialist for TAY and/or their families?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
______________________________________________________

18. What do you DISLIKE about being a Peer/Family Support Specialist for TAY and/or their families,
or where/when you feel stress in this role?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
______________________________________________________

19. How might your relationship with non-Peer/Family Support Specialist staff be improved? Do you
have any suggestions for improving the Peer/Family Support Specialist program for TAY and/or
their families at your facility and/or systemwide? If so, please provide your feedback in the space
below.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
______________________________________________________

20. Has working as a Peer/Family Support Specialist for TAY and/or their families affected your own
recovery/treatment management? Select one answer.
o Yes
o No (If no, skip to 22)
o I dont know

21. If yes, HOW has being a Peer/Family Support Specialist for TAY and/or their families affected
your own recovery/treatment management? Please explain.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
______________________________________________________

22. What new programs for TAY and/or their families would you like to see started if there were
resources available? Specifically, what programs would fill a gap in services for TAY? Please
describe the programs and why you feel these programs would be important, wanted, or necessary for
consumers.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
______________________________________________________

79
23. Where do you see yourself in 5 or 10 years within the Behavioral Health Services system of
care? Please describe.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
______________________________________________________

24. Do you have any other feedback that you would like to provide about being a Peer/Family
Support Specialist for TAY and/or their families? Please describe.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
______________________________________________________

Thank very much for helping us develop a better picture of Peer and Family Support Services for TAY.
We appreciate your feedback and your time. Have a great day! For any questions regarding this survey,
please feel free to contact us at: [evaluationmanageremail@organizationname.org]

80
MHSIP PSS Supplement Page

Some programs employ former or current consumers in the role of Peer Support Specialist, Peer Counselor,
or Peer Support Partner. A Peer Support Specialist/Peer Counselor/Peer Support Partner is someone who has
progressed in their own recovery from mental illness and can now offer to support consumers from the unique
perspective of someone with lived experience.

1. During the course of your care here, have you had any interactions with a Peer Support
Specialist/Peer Counselor/Peer Support Partner?
O Yes if yes, please ANSWER QUESTIONS 2, 3, 4, and 5.
O No if no, please skip to the next page.

2. If YES, what types of help did the Peer Support Specialist/Peer Counselor/Peer Support Partner
provide?
Please answer YES or NO to all of the following questions:
YES
NO
a. Provided advice or counseling
O
O
b. Helped me understand what resources were available
O
O
c. Helped me fill out paperwork
O
O
d. Helped me understand what was being asked of me by other staff
O
O
e. Served as a role model
O
O
f. Helped me set goals for my recovery
O
O
g. Helped me monitor my progress
O
O
h. Helped me navigate the mental health services system
O
O
i. Provided social support or reduced feelings of isolation
O
O
j. Other type of help (please fill in):
O
O
______________________________________________________________
______________________________________________________________

3. Please rate the following statements using the provided answer choices.
The Peer Support Specialist/Peer
Strongly
Strongly
Counselor/Peer Support Partner
Agree
Agree
Neutral Disagree Disagree N/A
a. Helped me believe I could recover
O
O
O
O
O
O
b. Understood my experiences
O
O
O
O
O
O
c. Was easier to speak with than other clinic staff
O
O
O
O
O
O
d. Was easier to speak with than my doctor
O
O
O
O
O
O
e. Provided helpful thoughts and insights
O
O
O
O
O
O
f. Made a difference in my recovery
O
O
O
O
O
O
g. Had experiences that were similar to my own.
O
O
O
O
O
O

4. What did you like best about meeting with a Peer Support Partner?
______________________________________________________________________________________
______________________________________________________________________________________

5. What did you dislike about meeting with a Peer Support Partner? Or was there another kind of help
you wanted the Peer Support Partner to provide?
______________________________________________________________________________________
______________________________________________________________________________________
***CSI County Client Number: __ __ __ __ __ __ __ __ __***

(Must be entered on every page)
81
YSS-Youth PSS Supplement Page

Some programs employ former or current consumers in the role of Peer Support Specialist or Peer Support
Partner. A Peer Support Specialist/Peer Support Partner is someone who has received mental health services
before but is not a therapist.

1. During the course of your treatment, have you met with a Peer Support Specialist/Peer Support
Partner?

2. If YES, what types of help did the Peer Support Specialist/Peer Support Partner provide?
YES
NO
O
O
b. Helped me understand what resources were available for me
O
O
c. Helped me to fill out paperwork
O
O
d. Helped me understand what was being asked of me by staff
O
O
e. Served as a role model for me
O
O
f. Helped me set the goals for my treatment
O
O
g. Helped me monitor treatment progress (determine if I was getting better)
O
O
h. Helped me understand the mental health services system
O
O
i. Provided social support or helped me feel less alone
O
O
j. Helped me get additional services
O
O
k. Other type of help (please fill in):
O
O
______________________________________________________________
______________________________________________________________

The Peer Support Specialist/Peer Support
Strongly
Strongly
Partner
Agree
Agree
a. Helped me believe I could recover
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
d. Was easier to speak with than my therapist
O
O
O
O
O
O
e. Provided helpful advice
O
O
O
O
O
O
f. Made a difference in my treatment
O
O
O
O
O
O
g. Had experiences that were similar to my own.
O
O
O
O
O
O

4. What did you like best about meeting with a Peer Support Partner?
______________________________________________________________________________________
______________________________________________________________________________________

5. What did you dislike about meeting with a Peer Support Partner? Or was there another kind of help
you wanted the Peer Support Partner to provide?
______________________________________________________________________________________
______________________________________________________________________________________

82
YSS-Family FSS Supplement Page

Some programs employ individuals as a Family Support Partner/Family Support Specialist whose child has
received mental health services before but is not a therapist.

1. During the course of your childs care here, have you had any interactions with a Family Support
Partner/Family Support Specialist?

2. If YES, what types of help did the Family Support Partner/Family Support Specialist provide?
YES
NO
O
O
b. Helped me understand what resources were available for my child
O
O
c. Helped me to fill out paperwork
O
O
d. Helped me understand what was being asked of me or my child by staff
O
O
e. Served as a role model
O
O
f. Helped me monitor my childs treatment progress
O
O
g. Helped me navigate the mental health services system
O
O
h. Provided social support or reduced feelings of isolation
O
O
i. Attended meetings (for example, IEP meetings) with me
O
O
j. Helped me get additional services for my child
O
O
k. Other type of help (please fill in):
O
O
______________________________________________________________
______________________________________________________________

The Family Support Partner/Family Support
Strongly
Strongly
Specialist
Agree
Agree
a. Helped me believe my child could recover
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
d. Was easier to speak with than my childs
O
O
O
O
O
O
therapist
e. Provided helpful thoughts and insights
O
O
O
O
O
O
f. Made a difference in my childs treatment
O
O
O
O
O
O
g. Made me feel better able to help my child
O
O
O
O
O
O

4. What did you like best about meeting with a Family Support Partner/Specialist?
______________________________________________________________________________________
______________________________________________________________________________________

5. What did you dislike about meeting with a Family Support Partner/Specialist? Or was there another
kind of help you wanted the Family Support Partner/Specialist to provide?
______________________________________________________________________________________
______________________________________________________________________________________


83

Tay Pss-Fss Evaluation Toolkit Draft 4-21-16

Încărcat de

Informații document

Descriere originală:

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Tay Pss-Fss Evaluation Toolkit Draft 4-21-16

Încărcat de

Drepturi de autor:

Formate disponibile

TRANSITION AGE YOUTH (TAY)

PEER AND FAMILY SUPPORT SERVICES

2. Complete the Feasibility Analysis Checklist (Pages 10-18)

4. Use the External Resources section (Pages 57-58)

5. Use the Evaluation Surveys (Appendix, pages 59-83)

6. Use the Double Data Entry Form (Supplementary Document)

7. Use the Report Template (Pages 37-55)

Toolkit Authors and Affiliations

Discussion: Interpretation of the Findings, Dissemination, and Next Steps ............................................ 53

TOOLKIT SUPPORT AND FEEDBACK .............................................................................................. 56

When is it appropriate to use this toolkit?

Why was this toolkit developed?

33% of county administrators described conducting TAY-specific process evaluations, compared to

What does this toolkit include?

1. Feasibility Analysis Checklist

4. Evaluation Survey Measures

Program Manager PSS/FSS Survey (Multiple pages)

5. Double Data Entry Form Template

How should the toolkit be used?

2. Complete the Feasibility Analysis Checklist (Pages 10-18)

4. Use the External Resources section (Pages 57-58)

5. Use the Evaluation Surveys (Appendix, pages 59-83)

6. Use the Double Data Entry Form (Supplementary Document)

7. Use the Report Template (Pages 37-55)

FEASIBILITY ANALYSIS CHECKLIST

Local TAY Programs and Consumers

75%-100%: Proceed; alternate strategies are optional, but may be helpful

TAY PSS/FSS Evaluation Feasibility Analysis Checklist

Actual number of programs that

2. How many of these programs

Actual number of programs exclusively

Actual number of TAY served: ______

Actual number of TAY

3. In total, how many TAY

5. How many of your TAY programs

6. In total, how many TAY receive

Actual number of TAY served by

Percent of TAY served by PSS/FSS:

Actual number of TAY PSS/FSS staff:

2. Data Collection, Entry, and Management

4. To what extent do you have access

7. How long does it take your

Actual number of staff available for

Length of time it usually takes your

3. Data Analysis and Reporting

3. How familiar are these staff

Actual number of staff with statistical

Briefly describe their experience:

4. What kind of software to conduct None, and

5. To what extent do you have

4. Evaluation & Quality Improvement (QI) Experience/Capacity

2. To what extent have these

3. How many staff do you have that

4. To what extent do you have a

5. To what extent have you been able

Advanced Briefly describe:

Actual number of staff involved in QI:

2. Data Collection, Entry, and

3. Data Analysis and Reporting

4. Quality Improvement (QI)

In each row, divide score by

OVERALL FEASIBILITY INTERPRETATION