100%(4)100% au considerat acest document util (4 voturi)
427 vizualizări56 pagini
Community-Based evaluation methods and data Analysis by amy Carroll. Quantitative vs. Qualitative Data Quantitative: Collected in the form of numbers or percentages. Qualitative: collected in form of words, concepts, themes, or categories. Secondary Data Activity Tracking Client Tracking Mapping / Photovoice Surveys Focus groups.
Community-Based evaluation methods and data Analysis by amy Carroll. Quantitative vs. Qualitative Data Quantitative: Collected in the form of numbers or percentages. Qualitative: collected in form of words, concepts, themes, or categories. Secondary Data Activity Tracking Client Tracking Mapping / Photovoice Surveys Focus groups.
Drepturi de autor:
Attribution Non-Commercial (BY-NC)
Formate disponibile
Descărcați ca PPT, PDF, TXT sau citiți online pe Scribd
Community-Based evaluation methods and data Analysis by amy Carroll. Quantitative vs. Qualitative Data Quantitative: Collected in the form of numbers or percentages. Qualitative: collected in form of words, concepts, themes, or categories. Secondary Data Activity Tracking Client Tracking Mapping / Photovoice Surveys Focus groups.
Drepturi de autor:
Attribution Non-Commercial (BY-NC)
Formate disponibile
Descărcați ca PPT, PDF, TXT sau citiți online pe Scribd
Amy Carroll April 22, 2005 Different Types of Data
Opinions, priorities Services or resources
provided Aspirations, motivations Resident utilization of services or resources Level of awareness, provided knowledge, attitudes or beliefs Numbers or rates of disease, illness, Behaviors, practices disability, injuries Assets, skills Sales transactions, Networks, associations purchases Needs, fears, problems, Policies concerns Pictures, other visuals Demographic Quantitative vs. Qualitative Data Quantitative: Qualitative: Collected in the form of Collected in the form of numbers or percentages words, concepts, themes, or categories Closed-ended questions Open-ended questions Answers who?, what?, when? and where? Answers how?, why? Can demonstrate cause Can provide richer, and effect more in-depth data Can “represent” a Can provide data in a population respondent’s own words Cannot collect new ideas or responses, only Can explore new ideas those considered ahead in a dynamic and of time unstructured way Common Community-Based Evaluation Methods • Secondary data • Activity tracking • Client tracking • Mapping/ Photovoice • Surveys • Focus groups • Key informant interviews Which of these methods collect quantitative data?
Which collect qualitative
data? Secondary Data • County health department • Vital statistics Birth & death certificates • City or county government • Schools (including school nurse records) • Hospital or clinic records • Local health promotion, health ed or advocacy agencies • Private foundations which fund community programs Universities or local community colleges • Local surveys or research efforts going on in your community?? • Data you already collect! Activity Tracking • Tracking and documenting program activity implementation • Tracking and documenting number of people reached • Process evaluation method, but important for outcome evaluation • May include: weekly/monthly program implementation log, sign-in sheets at classes or health fairs, counts of materials/ brochures distributed, etc. Client Tracking
• Tracking and documenting encounters
with clients; describing the encounter and client characteristics • More than just counting • Can be used for process and outcome evaluation • May include: outreach encounter form, home visit form, clinic intake form, etc. Mapping/ Photovoice • Data can also include maps and photos → don’t just have to be numbers or words • Maps and photos can provide visual imagery to give meaning to quantitative data • They can show graphically what cannot be described with numbers or words, such as the conditions of a neighborhood or program facilities, or a community’s proximity to park space or health services. • Make sure your maps and photos are clear and illustrate the points you would like to make Surveys • Surveys are composed of carefully crafted questions on a topic or issue for the purpose of revealing information about community residents or services they utilize. • The most commonly used survey methods are: 1. Mailed Surveys 2. Hand-out Surveys 3. Face to Face Surveys 4. Telephone Interview Surveys Planning A Survey
• Determine your purpose
• Determine if the survey data collection is appropriate • Select your target audience • Select a survey method • Design survey questionnaire • Pilot test survey • Publicize and distribute surveys Determine the Data You Need • Draft your research questions – what you want to learn from these interviews. • Identify the data you will need to collect in order to answer these questions. • The type of data you are looking for will help you identify the best persons in the community to survey, and the best survey questions to ask them. Target Population
The following are two steps in
determining your survey target population. 1) Define your target population -- identify the group of individuals from your community from who you want to collect information (single mothers, high school students, the uninsured, etc.) 2) Identify the geographic area of your target population.--Where is your group of interest located? What are the Select A Survey Method
Self Administered: These surveys are filled
out by respondents themselves without the assistance of trained interviewers. A. Mail Surveys B. Drop-off Surveys
Administered by Interviewer: These surveys
are filled out with the assistance of trained interviewers. A. Face to Face Surveys B. Telephone Surveys Design Survey Questionnaire • Your survey questions should be in direct relationship with what you want to learn from your target population. • Asking unrelated questions will interrupt the flow of the survey and tire out your respondents. • Be brief and strategic when designing questions! • Open-ended questions are great for answering “why” questions, but only include one or two. • Motivating people to fill out your survey will help you collect the data you need – this can include a benefits statement or incentives such as entry into a lottery or drawing. Pilot Test Survey
• It is always a good idea to pilot test
the survey with a small group of people from your target population. • The purpose for piloting the survey is to get feedback on the survey question and structure before implementing it. • This will ensure you identify potentially confusing, offensive, leading, or loaded questions from your survey. Publicize and Distribute Surveys Advertise your survey a couple of weeks before distribution in the community. Publicizing what you are trying to do and how the information collected will benefit the community may help increase your response rate. Create a survey distribution timeline and stick to it - postponing survey distribution date may limit the time you have set aside for data analysis. Advantages and Disadvantages of Surveys Advantages Disadvantages Data can be collected Survey instrument must from a lot of be carefully constructed respondents easier than to avoid leading any other method questions, and to make Can get a large enough sure the appropriate sample that can be responses are available representative of the Response rates can be larger population low for self-administered Findings can be surveys, especially generalized to the larger mailed ones population Response will be low if Can cover a lot of topics survey is too long Can easily compare Focus Groups
Focus group are a qualitative
research method designed to learn more about how people think, feel, or make decisions (attitudes, perceptions, opinions, experiences) through focused discussions. Focus Groups (cont.)
A series of discussions involving 8-12 people,
selected to share their perceptions of a defined topic Stimulates participants to share their opinions openly in a group discussion Captures rich data in participants’ own words Flexible to capture new ideas and issues Can be difficult to summarize and interpret results across groups Need to be creative when recruiting busy people Planning a Focus Group
• Determine the data you need
• Determine and select your target audience • Develop a focus group discussion guide • Schedule focus groups • Recruit focus group participants • Recruit moderator and note-taker • Plan and conduct focus groups Determine the Data You Need • Draft your research questions – what you want to learn from these focus groups. • Identify the data you will need to collect in order to answer these questions. • The type of data you are looking for will help you identify the best persons to recruit for your focus groups, as well as the best questions to develop for Target Audience
• To whom do you want to talk?
• What types of experiences and perspectives you are hoping to capture in your discussion? • Design and use a screening tool to recruit participants: ask questions that qualify people to participate. • It is important that you include participants who will feel comfortable talking to each other. • Participants should not know each How Many Focus Groups?
• Conduct 2-3 focus groups for each sub-
population of interest (i.e. gender, age, race/ethnicity, language, etc.) • 2-3 ensures that the perspectives and experience you capture are more representative of the larger population and not just by “chance”. • Length: 1.5 hrs.-2 hrs. • Focus Group Size: 6-12 participants Focus Group Discussion Guide • Discussion guide: An outlined script to help guide the moderator and focus the discussion • Organization: Introduction Purpose Ground rules Focus group questions Closing statements Discussion Guide Questions • Introductory questions: getting people to think about the general topic. • Transition questions: questions that get participants to logically transition between the introduction topics and the ideas contained in the key questions. • Key questions: 2-5 questions important to getting the information you have set out to collect. • Ending questions: questions that provide closure and reflection. • Summary question: note-taker summarizes the major themes heard throughout the discussion in a short oral summary (2-3 minutes), then will ask the participants if the summary covered all the major points. Scheduling Focus Groups
• Location: The focus groups should
take place in a convenient and accessible location. It is also important to select a neutral environment. • Time of day: most convenient for your target audience. Set time earlier than intend to start. • Incentives: Food or refreshments, money or gifts, drawings, child care, transportation, anything that is meaningful to those community members. Conducting & Moderating
• The discussion guide will help guide the
moderator with the questions you want answered. • Moderator Responsibilities: setting up ground rules, keeping the discussion focused and on track, drawing out comments from quiet participants, and asking for clarification or additional information from participants when needed. • Note-taker Responsibilities: takes comprehensive notes, operates audio equipment (tape-recorder), keeps track of time, handles environmental conditions and Skills of the Moderator
Extremely important to select moderators based
on skills and experience: • Comfortable/familiar with leading meetings or groups • Able to exercise mild control over the group • Able to guide conversation back on target • Maintain group enthusiasm and interest • Curious about the topic • Respects participants and their comments • Creates and maintains a comfortable environment • Adequate background knowledge • Good oral and written skills Advantages and Disadvantages of Focus Groups Advantages Disadvantages Flexible May be challenging to Captures rich, in- recruit participants depth data Need to schedule at Immediate results least 2-3 focus groups to capture diversity Encourages and stimulates individuals Difficult to generalize to share more openly results to the larger population because of Data can be combined small numbers of with quantitative data participants to provide a complete picture about an issue Difficult to compare results across groups Key Informant Interviews • Key informant interviews are qualitative, in- depth interviews with key informants • Key informants are individuals who know what is going on in the community (community leaders, professionals, residents, advocates, elected officials, doctors, etc.) • The following are two common techniques used to conduct key informant interviews: 1. Telephone interviews 2. Face-to face interviews Planning Key Informant Interviews • Determine the data you need • Determine target population and brainstorm possible key informants • Select key informants • Select interview technique • Develop an interview guide • Determine documentation method • Select designated interviewer(s) • Conduct key informant interviews • Compile and organize key informant interview data Determine the Data You Need • Draft your research questions – what you want to learn from these interviews. • Identify the information you will need to collect to answer these questions. • The type of data you are looking for will help you identify the best persons in the community to interview, and the best questions to ask them. Target Audience and Key Informants • Determine your target population (program clientele, racial/ethnic minority groups, adolescents, women, the elderly, HIV+ persons, etc.) • Brainstorm and create a list of possible key informants who are knowledgeable and closely involved with your target population. • Select key informants: • Who have first-hand knowledge about the target population, the program, and the issues you are trying to investigate. • Who represent different experiences, backgrounds, and viewpoints. Select Interview Technique
Telephone interviews (15-25 min):
• May be the most convenient and least time intensive way to interview busy key informants. • Disadvantage: not having the personalized interaction otherwise possible through a face-to- face interview.
Face-to-face interviews (20-30 min):
• Most frequently used format. But time intensive. • Advantages: provides a free-exchange of ideas and lends itself to ask more complex questions. Develop Interview Guide
Introduction- Introduce yourself and your
purpose and/or benefits statement. Key questions- Draft 5-10 questions important to getting the information you have set out to collect. Probing questions – Probing questions encourage participants to reflect more deeply on the meaning of their comments. Closing question—Provide an opportunity for the key informant to give any additional information or comments. Summary—Quickly summarize the major Determine Documentation Method Note-taking: manually write the key informant’s comments while conducting the interview, on a copy of the interview guide. Tape-recording: use a tape recorder to document what key informants say. This approach may allow the interviewer to freely engage in the conversation without worrying about note-taking. Conduct Interviews
• Thank respondents for their time and
stress the importance of the interview. • Interviewers should listen carefully for recurring and new opinions or beliefs. • In order to compare with other interviews and identify emerging themes, it is important to get answers from every person interviewed on key questions. Advantages and Disadvantages of Key Informant Interviews Advantages Disadvantages Rich data can be Selecting the “right” key gathered relatively informants may be difficult cheaply & easily so they represent diverse Allows interviewer to backgrounds and viewpoints establish rapport with the May be challenging to reach respondent, clarify and schedule interviews with questions, and draw out busy and/or hard-to-reach responses respondents Allows for discussion of Difficult to generalize results topic without group to the larger population dynamic of focus groups unless interviewing many Provides an opportunity key informants to build relationships with important Things to Consider Before You Select Your Methods • What data collection method(s) would best collect the type of data you need from your target population • What data collection method(s) would help you to best answer your research question(s)? • Think of your available resources: time, staffing, skills, funding, computer technology • Prior experience • Potential benefits • Potential challenges How to Understand the Data You Collect (Data Analysis) Why It’s a Good Idea to Plan Ahead for Data Analyses To determine if the data you collect are practical for analyses To review the appropriateness of your chosen data collection method(s) To inform the data collection instrument you develop To decide if you will need help with analyses 1: Check the Data
Make sure it’s all there
Make sure it makes sense Catch any mistakes that happened while filling out the survey or entering the data Perform data quality checks throughout your data collection process Common mistakes: – A missing or incomplete response – An impossible response – A response unrelated to the question – A response that contradicts an earlier response 2: Go Back to the Research Questions Be sure to analyze only the data that will help you answer your research question(s) Don’t get side-tracked into analyzing other pieces of data with your limited time and resources It is normal to collect more data than you will need for your evaluation You can return to any other interesting data after completing your evaluation 3: Reduce the Amount of Data Save your data in its original form so that you can return to it if you need to Look at it in small chunks at a time Eliminate any irrelevant data from your analysis Create summary documents to help you “eyeball” the data more easily to look for patterns or themes: – Numeric spreadsheet – Qualitative summary document 4: Analyze the Data Quantitative Data Analysis: Looking for Patterns in the Data Calculate averages – Used with “continuous” data: infinite number of values – Add numbered responses – Divide by number of responses – 2 + 4 + 6 + 10 = 22 22/4 = 5.5 Count frequencies – Used with “discrete” data: set number of response categories – Count number of responses – Number “yes”, number “no” – Number of choice “A”, Number of choice “B”, 4: Analyze the Data Quantitative Data Analysis: Looking for Patterns in the Data Calculate proportions – Is standardized – Equation: % = # of responses/ total # of respondents X 100 – Example: 45 females/ 100 respondents X 100 = 45% of the respondents were female Calculate rates – Similar to proportions; also standardized – Easier to compare – Equation: Rate = freq of event in pop/ total pop X 100,000 – Example: 15 cases of lung cancer/ population of 3,500 X 100,000 = 428 persons per 100,000 have lung cancer 4: Analyze the Data Quantitative Data Analysis: Looking for Patterns in the Data Compare averages, frequencies, proportions, rates Compare data from different populations: – County vs. County – County vs. State – State vs. US Compare data from different segments of a population: – Male vs. female – African American, white, Latino, Asian, American Indian, etc. – Children, teens, adults, seniors – Clientele of your program vs. those who are not 4: Analyze the Data Quantitative Data Analysis: Looking for Patterns in the Data
Present the data in different ways to see
additional patterns and relationships: – Chart/ table – shows averages, counts, proportions, or rates side-by-side – Pie graph – demonstrates percentages of the whole – Bar graph – compares quantities – Line graph – shows trends over time
Determine your findings
– Interesting results? – Interesting patterns or relationships? – These are your findings Exercise 5.4A (Using Step 5 of Performing a Community Assessment)
Look at sample survey and
sample spreadsheet in Appendix C Answer questions on page 51 and 52 Step 4: Analyze the Data Qualitative Data Analysis: Looking for Themes in the Data Qualitative data analysis can be deceptively trickier – Qualitative data is by nature “bigger” – Analysis requires more analytical thinking and interpretation Qualitative data is more open to “bias” – Numbers don’t lie – Analysts bring their values, assumptions and opinions – Analysts may think they “know” how people feel Include at least 2, preferably 3 people in each stage of data analysis to avoid bias Create as structured of a process as possible Step 4: Analyze the Data Qualitative Data Analysis: Looking for Themes in the Data
1. Read through all of the data at least twice
– Stay “close” or “grounded” in the data 2. Create categories – Focus group or interview questions – Or specific assessment health issues 3. List themes that emerge from discussion – Sub-categories – Ideas 4. Rank order according to frequency of appearance 5. Summarize discussion around each theme 6. Determine your findings Exercise 5.4B (Using Step 5 of Performing a Community Assessment)
Look at sample focus group
discussion transcript in Appendix C Answer questions on page 56 5: Verify Findings
You now have:
– Some patterns and relationships you found in your quantitative data – Some themes and categories you found in your qualitative data Based on the results you tallied and summarized, pull out the main findings of each method you used Verify these findings by re-tallying and re- summarizing the data to make sure you get the same results You can also verify your findings by comparing the results reached by 2 analysts independently 6: Interpret Findings and Draw Conclusions Determine what interpretations can be drawn from each finding Invite different perspectives from different Partners and community members, as this will make your conclusions stronger Are the results similar to what you expected? If not, discuss why you think they are different Brainstorm alternative explanations for your results to make sure you have considered all possibilities Make sure the conclusions answer the original assessment questions Draw conclusions and recommendations that can be shared with external audiences Exercise 5.6 (Using Step 5 of Performing a Community Assessment)