SHAHIZAH BINTI SHUKRI

Educational Multimedia 5th Colloquium

is PILOT STUDY
is a mini-version of a full-scale study or a trial run
done in preparation of the complete study
also called a feasibility study
specific pre-testing of research instruments, including
questionnaires or interview schedules (Polit, et al.&
Baker in Nursing Standard, 2002:33-44)
It is reassessment without tears (Blaxter, Hughes &
Tight, 1996:121), trying out all research techniques
and methods, which the researcher have in mind to
see how well they will work in practice.

OBJECTIVES PILOT STUDY
1.Pilot study is a small experiment designed to
test logistics
2. Gather information prior to a large study
3. Improve the actual studys quality and
efficiency
4.Reveal deficiencies in the design of a proposed
experiment or procedure and these can then be
addressed before time
5. A good research strategy requires careful
planning and a pilot study will often be a part of
this strategy
The Value of a Pilot Study
Welman and Kruger (1999:146) also listed the following
three values of a pilot study:
i. to detect possible flaws in measurement
procedures and in the operationalisation of
independent variables.
ii. to identify unclear or ambiguous items in a
questionnaire.
iii. give important information about any
embarrassment or discomfort experienced
concerning the content or wording of items in a
questionnaire- (the non-verbal behaviour)
Quotes concerning the value and goal of pilot
studies:
i. to see if the beast will fly
(De Vos, 2002:410)
i. reassessment without tears
(Blaxter, Hughes & Tight, 1996:121)
i. Do not take the risk. Pilot test first.
(Van Teijlingen & Hundley, 2001:2).
Pilot Study Quotes
Pilot studies should have a well-defined set of aims and objectives to
ensure methodological rigour and scientific validity.

Participants in an external pilot should not later be included in the
main study to make savings in recruitment, because then the decision
to proceed with the main study would not be made independently of
the results of the pilot study.

The analysis of a pilot study should be mainly descriptive or should
focus on confidence interval estimation.

Results from hypothesis testing should be treated as preliminary and
interpreted with caution, as no formal power calculations have been
carried out.

The temptation not to proceed with the main study when significant
differences are found should be avoided.
Recommendations
Pilot Study guidelines


Jumlah responden tidak
ditentukan dengan tepat,
dicadangkan sekurang-
kurangnya 25 orang, lebih baik
antara 50 75 orang.
Untuk kajian
baharu, lakukan
dua kali ujian
rintis.
Tidak boleh gunakan
kumpulan fokus
sebenar.
Size calculations may not be required for some
pilot or exploratory studies
A pilot study may be used to generate
information to be used for sample size
calculations
The sample for a pilot needs to be
representative of the target population
It should be sufficient to address the key
feasibility objectives
It should also be based on the same
inclusion/exclusion criteria
Sample Sizes for Pilot
Studies
1.Pilot study is done on a smaller scale.
Thus, actual results of the study may vary
from the results of pilot study.
2. A pilot study is usually carried out on
members of the relevant population, but not
on those who will form part of the final
sample
3. A pilot study is normally small in
comparison with the main experiment and
therefore, can provide only limited
information on the sources and magnitude of
variation of response measures

Limitation
VALIDITY !!!!
Validity is arguably the most important criteria for the quality of a test.

The term validity refers to whether or not a test measures what it intends
to measure.

There are several ways to estimate the validity of a test, including content
validity, construct validity, criterion-related validity (concurrent &
predictive) and face validity.


The question of validity is raised in the context of the three points made
above,
1. The form of the test
2. The purpose of the test
3. The population for whom it is intended.

Therefore, we cannot ask the general question Is this a valid test?. The
question to ask is how valid is this test for the decision that I need to
make? or how valid is the interpretation I propose for the test?
Validity is thus a requirement for both quantitative and qualitative
research
TYPES OF VALIDITY
Content related to objectives and
their sampling.

Construct referring to the theory
underlying the target.

Criterion related to concrete criteria
in the real world. It can be
concurrent or predictive.
Concurrent correlating high with
another measure already
validated.
Predictive Capable of anticipating
some later measure.

A correlation coefficient is a statistical summary of the relation between two
variables. It is the most common way of reporting the answer to such questions
as the following: Does this test predict performance on the job? Do these two
tests measure the same thing? Do the ranks of these people today agree with
their ranks a year ago?


According to Cronbach, to the question what is a good validity coefficient? the
only sensible answer is the best you can get, and it is unusual for a validity
coefficient to rise above 0.60, though that is far from perfect prediction.


All in all we need to always keep in mind the contextual questions:
what is the test going to be used for?
how expensive is it in terms of time, energy and money?
what implications are we intending to draw from test scores?
Reliability !!!!!!
How representative is the measurement?
Research requires dependable measurement.
Measurements are reliable to the extent that they are repeatable
and that any random influence which tends to make
measurements different from occasion to occasion or circumstance
to circumstance is a source of measurement error. (Nunnally,
1978)
Reliability is the degree to which a test consistently measures
whatever it measures. (Gay, 1987)
Errors of measurement that affect reliability are random errors and
errors of measurement that affect validity are systematic or
constant errors.
There are three major categories of reliability for most instruments: test-
retest, equivalent form, and internal consistency
Reliability
Test-retest the degree to which scores are consistent over
time. It indicates score variation that occurs from
testing session to testing session as a result of
errors of measurement.
Equivalent-Forms Used when it is likely that test takers will recall
responses made during the first session and
when alternate forms are available. Correlate
the two scores. The obtained coefficient is called
the coefficient of stability or coefficient of
equivalence.
Internal
Consistency
Determining how all items on the test relate to all
other items. Kudser-Richardson-> is an estimate
of reliability that is essentially equivalent to the
average of the split-half reliabilities computed for
all possible halves.
Two indexes of internal consistency:
Split half reliability
Coefficient alpha
INTER-RATER
RELIABILITY

the extent to which two or more individuals (coders or
raters) agree. Inter-rater reliability assesses the
consistency of how a measuring system is
implemented. Inter-rater reliability is dependent upon
the ability of two or more individuals to be consistent.
Training, education and monitoring skills can
enhance inter-rater reliability.

For example, when two or more teachers use a
rating scale with which they are rating the students
oral responses in an interview (1 being most
negative, 5 being most positive). If one researcher
gives a "1" to a student response, while another
researcher gives a "5," obviously the inter-rater
reliability would be inconsistent.
INTRA-RATER
RELIABILITY

type of reliability assessment in which the same
assessment is completed by the same rater on two
or more occasions. These different ratings are then
compared, generally by means of correlation. Since
the same individual is completing both assessments,
the rater's subsequent ratings are contaminated by
knowledge of earlier ratings.


Split half reliability

Splitting a test into two equivalent halves and
then assessing the consistency of the scores
across the two halves of the test.
Divide the test into halves and correlate the
scores from the two halves.
Compute the correlation between scores on
the two halves of the test using Spearman-
Brown formula.
The low correlation indicates that the test was
unreliable, a high correlation indicates that the
test was reliable.

Coefficient alpha

Lee Cronbach 1951) developed coefficient
alpha.. Alpha Cronbach
Coefficient alpha tells you the degree to which
the items are interrelated.
Rule of thumb:
At a minimum, greater than or equal to .07 for
research purposes and somewhat greater than
that value (e.g. .09) for clinical testing
purposes.


( ) correlation coefficients adalah seperti berikut:

0.00 hingga + 1.00 = pada asasnya

.60 hingga .70 = satisfied coefficients
.70 hingga .80 = stability coefficients
.80 hingga .90 = customary coefficients
.90 hingga .95 = sufficient coefficients
.80 hingga .90 = acceptable reliability
.90 hingga + 1.00 = very good reliability
.95 hingga + 1.00 = acceptable standardised test for internal
consistency
James Popham (1990), Modern Educational Measurement. A Practitioners
Perspective. 2
Nd Edition, New Jersey: Prentice Hall, Englewood Cliffs, h. 127.
Hair (2003)
What are some ways to improve validity?

Make sure your goals and objectives are clearly defined and
operationalized. Expectations of students should be written down.
Match your assessment measure to your goals and objectives.
Additionally, have the test reviewed by faculty at other schools to obtain
feedback from an outside party who is less invested in the instrument.
Get students involved; have the students look over the assessment for
troublesome wording, or other difficulties.
If possible, compare your measure with other measures, or data that may
be available.
RELATIONSHIP BETWEEN
VALIDITY & RELIABILITY
Validity and reliability are closely related.
A test cannot be considered valid unless the
measurements resulting from it are reliable.
Likewise, results from a test can be reliable
and not necessarily valid.
Additional information
1. Pearson product-moment correlation
2. Kesahan dan kebolehpercayaan dalam kajian kualitatif dan
kuantitatif
3. Comparison of values of pearsons and spearmans
correlation coefficients on the same sets of data