PII: sooo3-6870@6)ooo66-x

ELSEVIER

Applied Ergonomics Vol28, No. 3, pp 165-172, 1997 0 1997 Elsevier Science Ltd All rights reserved. Printed in Great Britain ooO3-6870/97 Sl7.00 + 0.00

Applying epidemiological principles to ergonomics: A checklist for incorporating sound design and interpretation of studies
Helen Heacock, Mieke Koehoorn and Joseph Tan
Department of Health Care and Epidemiology, University of British Columbia. Mather Building, 2804 Fairview Avenue, Vancouver, BC, V6T 123 Canada
(Received 2 January 1996; in revisedform 9 September 1996)

The primary purpose of this paper is to provide a checklist of scientific requirements necessary for the design of sound ergonomics studies. Ergonomics researchers will be able to use the checklist when designing a study and preparing it for publication. Practitioners can use the checklist to critically appraise study results, thereby having greater confidence when applying ergonomic recommendations to the workplace. A secondary purpose of the paper is to pilot the checklist on a sample of papers in the ergonomics literature and to assess its reliability. While there are checklists to assess the epidemiological rigour of studies, none have been adapted to address methodological issues in ergonomics. Two epidemiologists independently searched five ergonomics journals (Applied Ergonomics, Ergonomics, Human Factors, International Journal of Human-Computer Interaction and Journal of Human Ergology) for research studies on VDT use and visual function published between 1990 and 1995. Twenty-one articles were reviewed. Each paper was scored according to the checklist. Overall, the reviewers found that the articles did not consistently fulfill some of the checklist criteria. An insuffIcient sample size was the most serious omission. Inter-rater reliability of the checklist was excellent for 11 of 14 items on the checklist (Kappa >0.74), good for two items (Kappa between 0.40 and 0.74) and poor for one item. As ergonomics is gaining acceptance as an integral part of occupational health and safety, individuals in this field must be cognizant of the fact that study results are being applied directly to workplace procedures and design. It is incumbent upon ergonomists to base Geir work on a solid research foundation. The checklist can be used as a tool to improve study designs and so ultimately has implications for improving the fit between the worker and the work environment. 0 1997 Elsevier Science Ltd
Keywords: visual display terminals (VDTs), visual function, checklist, methodology, epidemiology

Introduction Impetus for this paper stems from articles questioning the state of ergonomics research (e.g. Meister, 1992; Oborne, 1991; Stock, 1991; Helander et al, 1984). These authors express concern over the lack of rigour and uniqueness shown in much ergonomics research. In a special Human Factors issue on measurement, Meister suggested the goal for human factors should be to develop a quantitative human performance data base and that the Human Factors Society should sponsor an annual conference devoted to human factors measurement (Meister, 1992). Oborne, addressing the 199 1 Ergonomics Society Conference, discussed a lack of specific ergonomics methodologies

and stated that ergonomists tend to carry out research using methods which reflect their own background and training. Oborne also commented on the need for more field-based studies as results are more generalizable to the end user population. He showed that of 72 studies published in Ergonomics in 1990, 75% were laboratory based and, of these, 80% involved subjects having no empathy for the research being performed (Oborne, 1991). With the introduction of visual display units (VDUs) in the early 197Os, a number of studies suggesting an association between eye discomfort and use of VDUs emerged. However, much of this research was based on subjective measures. Helander et al (1984) critically reviewed 28 studies of VDU use and visual discomfort. They found serious methodological

165

166

Applying epidemiological principles to ergonomics: H. Heacock et al

weaknesses, including a lack of control groups and biased samples in these studies, and concluded Lack of scientific rigor has reduced the value of many of these studies. Especially frequent were flaws in experimental design and subject selection, dependent and experimental settings and independent variables often made it difficult to generalize the results beyond the conditions of the particular study. In one of the first papers to critically assess the validity of studies on workplace ergonomic factors and the development of musculo-skeletal disorders of the neck and upper limbs, Stock found only two of 54 papers met rigorous criteria to demonstrate causality (Stock, 1991). Papers not meeting the criteria consistently lacked adequate controls, proper exposure measures and several did not clearly describe the outcome measure. Stock also reviewed previous reviews on the relationship ergonomic factors and between musculoskeletal disorders (Stock, 1991). She concluded that these reviews contained considerable methodological weaknesses (i.e. primary research questions and methods not clearly stated, validity of primary studies not assessed and variability of results not discussed). Hence, there appears to be a need to improve and standardize the quality of research methodology used in certain areas of ergonomics research. The purpose of this paper is to provide a checklist, based on criteria from the epidemiology field, for use by ergonomics researchers in the design and publication of research, and for use by ergonomics practitioners when assessing recommendations in ergonomics literature. A secondary purpose is to determine the reliability of the checklist. Epidemiology is the study of the distribution and determinants of disease frequency in human populations. It uses rigorous methodology to ascertain causality between exposure (e.g. VDT use) and health outcome (e.g. visual fatigue). For those wishing to further their epidemiology, understanding of Hennekens and Buring (1987) provide an excellent introductory text. In an attempt to substantiate concerns of those querying the rigour of ergonomics measures and methods, a search of the published literature on ergonomics methodology was undertaken to determine what information was available. A number of sources devoted to ergonomics methodology was found (e.g. Salvendy, 1987; Drury, 1990; Downton, 1991; Kantowitz, 1992 and Muckier, 1992). These resources discuss various types of ergonomics measures and methods including information on: sample size calculations, sample characteristics, sample and setting representativeness to enhance generalizability, the selection of measures, validity and reliability, and collecting and analyzing data. Hence, information on rigorous study design exists. To our knowledge, this information has not been synthesized into a userfriendly instrument to assess the validity of ergonomics research. As a result, ergonomics and epidemiological methodologies were coupled to develop the pilot checklist. We hope this checklist will be used like other ergonomics checklists-in this case, as an integral

component of ergonomics study design and as a tool in assessing the validity of published literature.

Methods
The checklist was designed by adapting questions from previous checklists developed for critical appraisal of scientific literature (Oxman et al, 1993; Guyatt et al, 1993; Schechter et al, 1991; Dawson-Saunders and Trapp, 1990; Stock, 1991). The checklist contains 14 questions in 11 key areas necessary for sound design and interpretation of studies. To assess the reliability of the checklist, the authors applied it to a sample of 21 studies on VDT use and visual function, and assessed inter-rater reliability by using the Kappa statistic, a frequently used measure in medical research. The Kappa coefficient is a calculation of agreement based on the proportion of yes/no responses in which two reviewers agree (Streiner and Norman, 1991). The procedure to include all relevant studies was as follows: A manual search of articles published between 1990 and 1995 on VDT use and visual function in the journals Applied Ergonomics, Ergonomics, Human Factors, Journal of Human Ergology and International Journal of Human-Computer Interaction, was undertaken. The studies that met the inclusion criteria (research articles for the defined domain published in ergonomics journals) were then independently evaluated using the checklist by two doctoral students in epidemiology who both have academic qualifications and/or work experience in ergonomics. The authors selected 14 items they deemed to be the most critical for reviewing articles. A description of the characteristics study necessar) to fulfill the epidemiological criteria is provided below.
Criteria to consider in the design and reporting of studies

The introduction section of a research paper should ideally introduce the reader to the study and include a review of the area of concern supported by the literature, followed by a brief description of the specific problem(s), the purpose/objectives of the study, including the research question(s), the theoretical framework and finally the research hypothesis(es). It is not always feasible to provide all of this information in a journal article. However, the authors feel that the researcher must demonstrate that a gap exists in the literature and then clearly state how the study will attempt to address this gap by describing the relationship between the outcome and exposure variable to be studied.
Methods. (i) Study design and methodology. The study design is the overall plan of action for the proposed study and should validly evaluate the research question(s) given the constraints of the study. The study methodology is a road map of explicit instructions for the study design. After reading the methodology, the reviewer should feel like he or she could attempt to carry out the study themselves. Practitioners should feel confident in applying the results in the workplace and researchers should be confident in replicating or expanding upon the study design and methods.

Introduction.

Applying epidemiologicalprinciples to ergonomics: H. Heacock et aa (ii) Variables and procedures. In the methodology section, both exposure/independent variables (e.g. luminance) and outcome/dependent variables (e.g. visual strain) as well as their units of measurement should be documented in such a manner that the reader is confident of the suitability (i.e. reliability The ergonomic and validity) of these measures procedure(s) should be clearly described as well as its potential ability to show change, for example, adjusting luminance levels has the potential to reduce eye strain. (iii) Study population. The study population must be explicitly defined. The following are points to be included in describing the study population: population characteristics (e.g. age, gender, task experience), inclusion/exclusion criteria, a description of the comparison group(s) and how it was obtained. Comparison groups (or in the case of multivariate analyses, a group having little exposure) are necessary in research to describe a baseline with which the effects of an independent variable can be assessed. Sometimes a comparison group will consist of the experimental group tested before an intervention or exposure to an explanatory variable. Such pre-post measurements must be compared to another group who have not been exposed to the intervention, exposed to a different intervention or exposed to different levels of the explanatory variables. In some cases, it is appropriate for the same subjects to serve as their own controls. In other words, they participate in both the intervention and non-intervention groups at different points in time throughout the study (i.e. repeated measures or selfcontrol design). Sample size calculations are vital to epidemiological research as sample size determines power (the ability of a study to detect a difference if one truly exists). Comparisons of back pain outcomes between two groups each comprising five individuals does not provide a very strong (i.e. powerful) conclusion. As the power or strength of the conclusion diminishes so too does the validity of the results. Tables have been developed to assist researchers determine the sample size required to detect true differences (see Cohen, 1988). All of these factors should leave the reader feeling that the study population is appropriate for the study design, representative of the broader population and sufficiently large to support statistical analyses. (iv) Data collection. Data collection should be described in terms of who collected the data, what data were collected, where it was collected, and what checks and balances were in place to ensure reliable, valid and unbiased results. Ensuring reliable, valid and unbiased results requires researchers to provide some justification for the ability of their procedures or instruments to truly and consistently measure all variables under investigation.

167

appropriate test(s) given sample size, number of study groups and the type of outcome measure. For example, if sample sizes are small, e.g. less than 30, and outcomes are not normally distributed, parametric statistical tests (e.g. t-test, coirelation coefficient) may not be robust enough and hence, non-parametric tests (e.g. Wilcoxon rank sum, Spearmans rho) can be used. When determining the appropriate sample size for multiple regression analyses, statisticians recommend using 10 times as many subjects as there are independent variables (Dawson-Saunders and Trapp, 1990). Readers can refer to statistical decision trees to assess the appropriateness of statistical tests within studies.

Results
This section contains results of the research that relate to the research question. Findings are presented as descriptive statistics (e.g. means, medians, ranges, standard deviations, proportions), often in tabular format and as results of statistical tests. The p values and corresponding confidence intervals should be presented.

Discussion and conclusion

The discussion and conclusions section should provide answers to questions posed in the Introduction and relate findings to data presented in the Results section. The authors should address consistency of results or lack thereof in terms of previous studies. The investigators should also point out limitations of their study methods and discuss how these limitations affect the direction of the results. The inability to measure a potentially confounding variable or to rigorously control for a confounding variable is a frequent practical limitation of studies. Such limitations may erroneously inflate a result or may mask the true effect of an explanatory variable on the outcome of interest. It is helpful for readers when interpreting the results of a study to know the limitations and to be able to make a judgment about how seriously the limitations affect the findings. Lastly, recommendations and practical significance of the findings should be presented as well as how future research can address unanswered questions. From the above discussion, 14 questions have been selected from 11 key areas as most important in designing and reporting research studies, and thus make up the assessment checklist. Each of these areas is weighted equally in terms of importance for methodological rigour and is assigned a value of 1 point in the checklist, with one exception; key area number 9 (Appropriateness of Statistical Methods) has been assigned a value of 2 because this area is often neglected in ergonomics research and it is vital to the valid interpretation of results. In three key areas (Representativeness, Limitations and Implications) the weighting has been divided into two one half parts because there are two sub questions. In two other key areas (Instrument Reliability/Validity and Statistical Test) it is recognized that there might be partial adherence to the criteria and therefore values of O-5 or I.0 might be allotted.

(v) Statistical analyses. Also included in the section on methods should be a description of the statistical test(s) used. The statistical test(s) should be appropriate for the type of data collected, the population studied and suitable to answer the research question. Most statistical texts have decision trees for selecting the

168

Applying ep~demioiogic~i principles to ergonomics: H. Heacock

et al

GAP IN THE LITERATURE (1) Does the study identify a gap in the existing literature. In other words, does the review of the literature suggest any of the following: there is limited information available on the topic a new procedure or tool will look at a previously explored question improvements in study design or methods. Yes (1) No~Insu~cient RESEARCH QUESTION info~ation (0)

doing similar work to the experimental group but not using VDTs. Yes (1) No/Insufficient SAMPLE SIZE/POWER (8) Was sample size or power discussed? Comparisons of visual fatigue Example: outcomes between two groups each consisting of live individuals does not provide a very strong (i.e. powerful) conclusion. Yes (1) No/Insufficient STATISTICAL TEST information (0) information (0)

2)

Is the research question clearly stated? In other words, is there a statement indicating that the study will examine, investigate or measure an stated exposure between a association (independent variable) and a stated outcome (dependent variable)? Example: The aim of the proposed study is to examine the relationship between prolonged VDT use (hours per day) and visual strain. information (0)

Yes (1) No/Insufficient METHODOLOGY

(9) Do the authors state the statistical test(s) used and the level of significance/confidence interval (CI) achieved? Example: A paired t-test showed a significant relationship between the expe~mental and control group, p < 0~001. Statistical test: Yes (0.5) No/Insufficient information (0) p value or CI: Yes (O-5) No/Insufficient information (0) LIMITATIONS

(3) After reading the methodology section, could you repeat the study with confidence? Yes (1) No/Insufficient MEASUREMENT INSTRUMENTS information (0)

(10) Does the paper identify limitations and biases of

design/methodology? Example: The study design did not take presbyopia into account and the experimental group was, on average, five years older than the control group so results may reflect, in part, the process of aging on certain visual fatigue indices. Yes (O-5) No/Insu~~ient information (0) (11) Do the authors discuss impact of limitations on the results? Example: In the preceding example, the study may result in higher scores on the visual fatigue indices in the ex~~rnent~ group possibly leading a statistically significant difference when one does not truly exist. Yes (0.5) No/Insuflicient APPROPRIATENESS METHODS information (0)

TECHNIQUES/

:4) Do the researchers state whether all measurement te~hnique(s}~instr~ents are valid or reliable, or do they provide additional info~ation (i.e. references or product/manufacturing names) so the reader can investigate further the validity and reliability of study instruments and techniques? validated Using a previously Example: questionnaire by Karasek et al. Muscle balance View was using a Keystone measured Telebinocular (Mast/Keystone, Ophthalmic Davenport Iowa). Yes (1) No/Insu~cient information (0)

OF STATISTICAL

REPRESENTATIVENESS/GENERALIZABILITY (5) Is the study population representative user population? Example: In a study and visual function, clerical workers data entry work as opposed to using do the same task. Yes (0.5) No/Insufficient information (0) of the endof VDT use performed students to

(6) Is the

study setting representative of the workplace to which the results will be applied? Example: Visual discomfort experienced by VDT operators was assessed using a questionnaire administered at the worksite following a regular shift of VDT work. information GROUP (0)

(12) Do the authors utilize an appropriate sample size or do they utilize appropriate statistical tests for smaller sample sizes according to any one of the following? Utilize a study population N230; Conduct non-parametric statistical tests for N < 30; Conduct parametric statistical test for N< 30 but explain that the sample is l~ormally dist~bute~, Conduct multivariate tests, including regression, with 10 times as many subjects as there are independent variables. Reference: Dawson-Saunders and Trapp, 1990. Basic and Clinical Biostatistics, Appleton and Lange, Conn~ticut. Yes (2) No/Insufficient information (0) Recommendations and opportunities for future research are weak based on small sample sizes and/or inappropriate statistical analyses. If you respond No~Insufflcient information to question 10, do not proceed to 13 and 14.

Yes (O-5) No/Insufficient CONTROL/COMPARISON

(7) Is there an appropriate ~ontrol/comparison group (versus a pre-post comparison using the same group)? Example: A group of clerical workers

Key Area - Gap in Literature: Does the study identify a gap in the existing literature (limited information on topic,
Is the research question clearly stated (i.e. the study will examine, investigate or

Key Area - Research Question:

are valid/reliable OR do they provide information so the reader can investigate the validity and reliability of the ulation representative of the end-user population?
Is the study setting representative of the workplace to which the results wili be

r some but not a

Key Area - Representstiveness:

Group: Is there an appropriate control/comparison group (verses a pre/post single

D No/Insufftcient Information

Key Area - Control/Comparison

Key Area - Sample Sizez Is there a statement about the sample size or power of the study?

Key Area - Statistical Test(s): Do the authors state the statistical test used and the level of significance or confidence levels achieved?

Key Area - Limitations/Bias: Do the authors discuss the limitations or biases of the study design and/or methodology?

direction of effect on results)?

Utilize a study population (n) z 30 or; Conduct non-parametric statistics tests for n < 30 or;
If N~nsuff. info., DO NOT

Key Area - Implications: Do the authors discurq recommendations cf the findings for workers? ONLY ANSWER QUESTION 13 IF YOU RESIONDED YES TO QUESTION 11

~1 No/Insufftcient Information I 0 Yes

0 0.5

Key Area - Implications: Do the authors ident@ opportunities for further research based on the current study?

Figure 1 Checklist to assess the scientific rigour of ergonomics studies

170 IMPLICATIONS

Applying epidemiological principles to ergonomics: H. Heacock et al

(13) Do the authors discuss recommendations or the practical implications of the findings for workers and the workplace? Example: The maximum eye movement value of 48% should be a useful ergonomic index to evaluate quantitatively visual work and to improve comfort in the design of a workstation. Yes (0.5) No/Insufficient information (0) (14) Do the authors indicate where and how future research can assist in answering unanswered questions? Example: The Zone of Clear Single Vision (ZCSV) seems to be a sensitive indicator of eye-muscle work load in this experiment. ZCSV is recommended for future research both in terms of convenience and sensitivity. Yes (O-5) No/Insufficient Total points: 12 information (0)

The version of the checklist for use by ergonomists can be found in Figure I.

Results
The scores for each item on the checklist reviewed by the two epidemiologists were averaged and totals are as follows: ~6 69 >9 points: points: points: eight papers seven papers six papers

and statistical procedures. Most, but not all, used valid and reliable measurement techniques, and most considered the limitations of the study. The major weaknesses in these papers tended to be a weak description of the research under investigation, and a reliance on study populations and settings not representative of the workforce or workplace. 6 and less: Papers with this score were considered weak academic papers. The major weakness of this group was the reliance 011 small sample sizes with little consideration given to appropriate statistical analyses. Most papers failed to clearly identify a research need or to clearly state the relationship under investigation. limitations failed to describe and Many recommendations. Table 1 indicates the number of times agreement and disagreement between the two independent observers occurred, as well as inter-rater reliability as assessed by Kappa scores. A Kappa score of 1 indicates complete agreement between the observers, either they both agree the response to the question is yes or they both agree the response to the question is no. According to the Landis and Koch (1977) interpretation of Kappa, inter-rater reliability is excellent for 11 of the 14 questions (ranging from 75 to lOO%), good for two questions (ranging from 53 to 64%) and poor for one (Kappa=36%). Items on the checklist that showed weaker reliability were whether or not a gap in the literature existed, whether the research question was clearly stated and whether the direction of bias in the limitation(s) was recorded.

Discussion
Reliability of the checklist

The authors offer the following interpretation of scores: > 9-12: Papers that scored between 9 and 12 inclusive are strong academic papers. They tend to demonstrate a clear research need, designed with valid and reliable measurement techniques. Sufficiently large study populations (N> 30), including an appropriate comparison group, and suitable statistica! analyses were used. Finally, thoughtful discussions of the studys limitations, biases and opportunities for future research were included. > 69: Papers scoring more than 6 and up to 9 points were moderately strong papers. All used a large study population with appropriate comparison groups

The checklist of 14 criteria presented in this paper should serve as a useful reference for ergonomists wishing to produce rigorously designed research that is both valid and publishable. As a tool for both ergonomics practitioners and researchers, the checklist showed excellent inter-rater reliability for 11 of the 14 questions according to the interpretation of Landis and Koch (1977). Three of the questions (gap in the literature, research question and limitations) showed weaker inter-rater reliability. In developing the checklist, the authors attempted to cover the criteria necessary to ensure scientific rigour. Some of these are easier to quantify than others, such as whether or not a

Table 1 Level of agreement between two independent observers and Kappa statistic for the 12 studies reviewed Criteria (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) Gap in literature Research question Methodology Reliability/validity Study population Study setting Control group Sample size/power Statistical test Limitation(s) Direction of effect TV>30 Recommendation Future research interpretation according Both agree (l/l) 12 14 17 15 13 12 10 2 17 13 15 3 4 8 to Landis Both agree (O/O)
0

Disagree (l/O)
0

Disagree (O/l) 6 0 0 0 0 1 0 0 0 0 0 0 0 0

Kappa 0.36 0.60 0.83 1.0 1.0 0.80 0.81 1.0 1.0 1.0 1.0 0.59 0.85 1.0

4 3 6 8 7 9 19 4 8 6 15 8 5 and Koch (1977): Excellent: Kappaz0.75;

3
0 0

Kappa

Good:

Kappa

0.4W.74;

Poor: Kappa

< 0.40

Applying

epidemiological

principles

to ergonomics:

H. Heacock

et al

171

p value was provided, and whether or not an instrument was described in terms of its validity and reliability. Others are not as easy to assess quantitatively and hence showed weaker inter-rater reliability. Resolving this issue is difficult because the questions often involve subjective decision making around differences in presentation styles of various authors. For example, consideration was given to an exact definition of a clearly stated research question (i.e. an explicit statement such as The purpose/ objective/goal of this study is...). This definition unnecessarily penalizes studies that present their research question in a different manner. As a pilot of the instrument study, this initial application identified areas for further refinement. Applying the checklist to ergonomics papers on VDT use and visual function A purpose of human research is to obtain information from a study population that can be applied to a larger population. In the context of ergonomics, one goal is to design safe and healthy working environments for working populations. It is imperative that studies be designed and carried out using criteria that will result in valid and reproducible results. The pilot study demonstrated that a checklist can be used to identify methodological issues in the design and interpretation of ergonomic studies. The major weakness in the papers reviewed was an insufficient sample size. In the past decade, the issues of power and sample size calculations have come to the fore in the design of scientific studies. Only two of the reviewed papers mentioned sample size (in no instances did the authors indicate that they performed sample size or power calculations). The study population ranged in number from five to 34 1, with seven studies having fewer than 10 subjects. While small studies have their merit as pilot studies, results based on such samples have limited use. An ergonomic design, such as appropriate VDT contrasts, based on the results of 10 subjects may not be generalizable to a larger end-user population. Such results should be interpreted with caution. Recommendations Based on this pilot study we recommend further testing and refinement of the checklist by practitioners and researchers in different domains of ergonomics. Future assessments involving more than two reviewers can be performed using a Kappa for multiple assessors (Fleiss, 1971). Every effort should be made by those assessing the checklist to identify as many research studies as possible to ensure generalizability of results. We welcome feedback on the content and application of this checklist.

ergonomics research studies and in I Gtical appraisal of the literature. From the 21 papers reviewed according to the checklist we found that certain criteria required to fulfill epidemiological rigour are consistently neglected, in particular, sample size/power and appropriate statistical tests for the population being studied. The authors believe that further refinement and application of this piloted checklist will enhance the validity and reproducibility of ergonomics research.

Acknowledgements
Part of this research was supported by grants from the British Colombia Health Research Foundation and The National Health Research Development Programme.

References
Cohen, J. 1988 Statistical Power for the Behavioural Sciences, 2nd edn, Lawrence Erlbaum Associates, New Jersey, pp 5266 Dawson-Saunders, B. and Trapp, R. 1990 Basic and Clinical Biostatistics, Appleton and Lange, pp 274276 Downton, A. 1991 Evaluation techniques for humancomputer systems design in Downton, A. (ed) Engineering the HumanComputer Interface, McGraw Hill, NY, pp 325-355 Drury, C. G. 1990 Designing ergonomics studies and experiments in Wilson, J. R. and Corlett, E. N. (eds) Evaluation of Human Work. A Practical Ergonomics Methodology, Taylor and Francis, London, pp 101-130 Fleiss, J.L. 1971 Measuring nominal scale agreement among many raters Psycho1 Bull 76, 378-383 Guyatt, G.H., Sackett, D.L. and Cook, D.J. 1993 Users guide to the medical literature Il. How to use an article about therapy or prevention. A. Are the results of the study valid? JAMA 270(21), 25988260 1 Helander, M.G., Billingsley, P.A. and Schurick, J.M. 1984 An evaluation of human factors research on visual display terminals in the workplace Human Factors Review 3, 555129 Hennekens, C. H. and Buring, J. E. 1987 Epidemiology in Medicine, 1st edn, Little and Brown, Boston, pp 1639 Kantowitz, B.H. 1992 Selecting measures for human factors research Human Factors 34(4), 3877398 Landis, J.R. and Koch, G.G. 1977 The measurement of observer agreement for categorical data Biometrics 33, 159-174 Meister, D. 1992 Special issue preface Human Factors 34(4), 383385 Muckier, F.A. 1992 Selecting performance measures: objective versus subjective measurement Human Factors 34(4), 441-455 Ohorne, D. 1991 Tipping the balance towards ergonomics Proceedings of the 1991 Ergonomics Society Conference, pp 2-12 Oxman, A.D., Sackett, D.L. and Guyatt, G.H. 1993 Users guide to the medical literature 1. How to get started JAMA 270(17), 2093-2097 Salvendy, G. 1987 Handbook of Human Factors, John Wiley, New York Schechter, M. T., Lehlanc, F. E. and Lawrence, V. A. 1991 Critical appraisal of published research in Mulder, D., McPeek, B., Troidl, H., Spitzer, W., McKneally, M., Weschler, A. and Balch, C. M. (eds) Principles and Practice of Research: Srraregy for Surgical Investigators, 2nd edn, Springer, pp 81-87 Stock, S. 1991 Workplace ergonomic factors and the development of musculoskeletal disorders of the neck and upper limbs: a metaanalysis Am J Ind Med 19, 87-107 Streiuer, D. L. and Norman, G. R. 1991 Health Measurement Scales: A Practical Guide to Their Development and Use, Oxford University Press, pp 9495 Journal articles on VDT use and visual function reviewed: Berqvist, T.U., Knave, B., Voss, M. aud Wilbom, R. 1992 A longitudinal study of VDT work and health Int J HumanComputer Interaction 4(2), 197-219 Collins, M., Brown, B., Bowman, K. and Carkeet, A. 1990 Workstation variables and visual discomfort associated with VDTs Applied Ergonomics 21(2), 157-161 Collins, M., Davis, B. and Goode, A. 1994 Steady-state accommodation response and VDT screen conditions Applied Ergonomics 24(5), 334338

Conclusion
The checklist developed to assist ergonomists with designing scientifically sound studies showed excellent inter-rater reliability in 11 of 14 questions. Three questions which involved more subjective decision making proved difficult to score and resulted in weaker inter-rater reliability. The authors feel this piloted instrument can be used as a tool in both the design of

172
Gallimore,

Applying

epidemiological

principles

to ergonomics:

H. Heacock

et al

J. and Brown, M. 1993 Effectiveness of the C-sharp: reducing ergonomics problems at VDTs Applied Ergonomics 24(5),

321-336

Gratton, I. 1990 Change in visual function and viewing distance during work with VDTs Ergonomics 33(12), 1433-1441 Ishikawa, S. 1990 Examination of the near triad in VDU operators
Ergonomics 33(6), 187-198 Itoh, Y., Hayashi, Y., Tsukui, I. and Saito, S. 1990 The ergonomic

Saito, S., Taptagaporn, S. and Salvendy, G. 1993 Visual comfort in using different VDT screens Int J Human-Computer Interaction 5(4), 3 13-323 TyrelJ, R.A. and Leibowitz, A. 1990 The relation of vergeance effort to reports of visual fatigue following prolonged near work Human
Factors 32(3), 341-357 Udo, H., Tanida, H., Itani, T., Otani, T., Yokota, Y., Udo, A., Omoto, Y., Tuboya, A. and Yokoi, Y. 1991 Visual load of 20, 109-121 Watten, R. and Lie, I. 1992 Time factors in VDT-induced myopia and visual fatigue: an experimental study J Human Ergology 21, 13-20 Watten, R., Lie, I. and Birketveldt, 0. 1994 The influence of long-

evaluation of eye movement and mental workload in aircraft pilots Ergonomics 33(6), 719-733 Jasd&ki-Knua, W. 1991 Eyestrain in VDU users: viewing distance and the resting position of ocular muscles Human Factors 33(l), 69-83 Kamienska-Zyla, M. 1993 Ergonomics evaluation of the work of VDT operators in Poland Applied Ergonomics 24(6), 432433 Lie, I. and Watten, R. 1994 VDT work, oculomotor strain, and subjective complaints: an experimental and clinical study
Ergonomics 37(8), 1419-1433

working with visual display terminals-Introduction newspaper editing and visual effect J Human

of VDT to
Ergology

Pastoor. B. 1990 Legibilitv and subiective preference for color combination in text &man Factors 3i(2), 15%171 Saito, S. 1992 Does fatigue exist in a quantitative measurement of eye movements? Ergonomics 35(5/6), 607-615 Saito, S., Saito, S. and Ohkubo, T. 1991 An evaluation of VDT operations through the two dimensional analysis of fixation points
Int J Human-Computer Interaction 3(3), 245-252

term visual near-work on accommodation and vergence: a field study J Human Ergology 23,21-39 Yaginuma, Y., Yamada,--H. and Nagai, H. 1990 Study of the relationship between lacrimation and blink in VDT work
Ergonomics 33(6), 799-809

Yeow, P. and Taylor, S. 1990 The effects of long-term VDT usage on the nature and incidence of asthenopic symptoms Applied
Ergonomics 21(4), 285-293 Zhu, Z. and Wu, J. 1990 On the standardization of VDTs proper and optimal contrast range Ergonomics 33(7), 925-932