Clin. Otolaryngol.

2004, 29, 538544

The reliability and sensitivity to change of acoustic measures of voice quality

P. N . C A R D I N G , * I . N . S T E E N , A . W E B B , * K . M A C K E N Z I E , I . J . D E A RY & J . A . W I L S O N *
*Department of Otolaryngology, Head and Neck Surgery, University of Newcastle, UK, Centre for Health Services Research, University of Newcastle, Newcastle - Upon - Tyne, UK, Department of Otolaryngology, Head and Neck Surgery, Glasgow Royal Inrmary, Glasgow, UK, and Department of Psychology, University of Edinburgh, Edinburgh, UK
Accepted for publication 1 October 2003

carding p.n., steen i.n., webb a., mackenzie k., deary i.j. (2004) Clin. Otolaryngol. 29, 538544

&

wilson j.a.

The reliability and sensitivity to change of acoustic measures of voice quality

This study aimed to evaluate the reliability and sensitivity to change of three commonly used acoustic parameters as measured by the Multi-Dimensional Voice Programme (MDVP); jitter, shimmer and noise-to-harmonic ratio. A total of 231 subjects voices were recorded and analysed. The sample comprised 145 dysphonic patients who received intervention (surgery or voice therapy), 36 dysphonic patients who received no intervention, and 50 non-dysphonic (normal) subjects. All voices were recorded and analysed on two occasions (before and after treatment, or testretest assessment) using a standard procedure. These data were analysed using standard psychometric procedures for assessing reliability and responsiveness. The acoustic analysis measures demonstrated poor to moderate reliability and effect size with respect to their sensitivity to change. Caution should be exercised in the injudicious use of computer-based acoustic analysis systems as an isolated measure of voice outcome in any clinical trial of interventions aimed at improving voice quality. Keywords voice disorders dysphonia voice quality acoustic measurement perturbation

The need for instrumental objective assessment of voice quality can be seen in the increasing use of acoustic analysis methods for clinical diagnosis, outcome measurement and research. Acoustic analysis of the vocal signal has been applied to almost every area of voice care including the evaluation of surgical procedures,15 voice therapy,6,7 radiotherapy,8,9 medical therapy, 1012 screening of laryngeal diseases,13,14 and vocal pathology differential diagnosis.1517 However, there remains no standardization of technique methodology and considerable variability in which acoustic parameters to measure (as demonstrated by the references above). Even in one computer analysis system (e.g. The Computerized Speech Laboratory), the clinician is faced with a perplexing number of parameters from which to select. One useful means of informing the choice would be robust evidence of a parameters reliability and sensitivity to change as applied to a typical voice pathology clinical
Correspondence: Paul Carding, Department of Otolaryngology, Head and Neck Surgery, Freeman Hospital, Newcastle upon Tyne, NE7 7DN, UK (e-mail: paul.carding@ncl.ac.uk).

population. However, this information is surprisingly sparse and incomplete.1820 This study aimed to evaluate the discriminatory power, reliability, and effect size of the sensitivity to change of three commonly used acoustic parameters as measured by the Multi-Dimensional Voice Programme (MDVP); jitter, shimmer and noise-to-harmonic ratio (NHR).

Method
The patient sample comprised three cohorts: 1. A total of 145 subjects complaining of hoarseness and attending two university otolaryngology clinics (Newcastle and Glasgow) who received an intervention [90 had speech and language therapy (SLT) and 55 had surgery]: Their voice was assessed on two occasions, once before and once after the intervention. Fourteen of these subjects also participated in the testretest assessment of the acoustic measures. This retest was performed within 2 h following the initial assessment and prior to any intervention.
2004 Blackwell Publishing Ltd

538

Change of acoustic measures of voice quality 539

2. A total of 36 dysphonic subjects who received no intervention: All these subjects participated in the testretest assessment of the acoustic measures; the two assessment recordings were made within 2 h of each other to reduce the effect of the inherent variability of voice quality in dysphonic patients. 3. A total of 50 subjects with a normal voice: The voices of these subjects were also recorded twice within 2 h of each other. Sample demographics are given in Table 1. All patients were recorded in a sound-proof room (ambient noise <40 dB) using a Sony DCT690 DAT recorder (sampling frequency 48 kHz; Sony Corporation, Allandale, NJ, USA) and a Sony electret microphone (omnidirectional and sensitivity of 60 dB). The microphone-to-mouth distance was 10 cm and careful positioning minimized aerodynamic noise. No effort was made to control the absolute sound pressure levels of phonation. However, patients were instructed to produce a prolonged /a:/ sound at a comfortable pitch and attempting to keep the sound as stable as possible for 6 s (or as long as they could manage). Patients were allowed to practice the task several times before the recorded sample /a:/ was performed. A total of 86 subjects (36 non-intervention dysphonics and 50 normal subjects) were also recorded a second time within 2 h of the rst recording following an identical recording procedure. These data were used for testretest reliability analysis. The whole vowel sample was digitized into a computer and the exact middle second of the prolonged vowel sample was identied and analysed using the MDVP of The Computer Speech Laboratory Model 4300B (Kay Elemetrics, Lincoln Park, NJ, USA). The analysis sampling rate was set at 25 kHz. Measures of frequency perturbation (jitter), amplitude perturbation (shimmer) and NHR were noted. These
Table 1. Sample demographics Cohort Subjects with abnormal voice Intervention cohort Sample size Mean age (range) Sex [n (%)] Male Female Diagnosis [n (%)] Organic Non-organic Movement disorder Systemic disease Not recorded Treatment [n (%)] Voice therapy Surgery 145 52.5 (1787) 47 (32.4) 98 (67.6) 46 57 23 18 1 (31.9) (39.6) (16.0) (12.5)

measures represent three of the most commonly used acoustic parameters from the MDVP. It was recognized that some voices may be too severely dysphonic (too aperiodic) to produce meaningful acoustic analysis values (i.e. type III acoustic voice signals21). One purpose of this study was to determine the extent to which acoustic analysis could be applied to a range of dysphonic voices commonly encountered in clinical practice. data analysis The distributions of the acoustic indices were assessed graphically and through consideration of summary statistics. Discriminatory power of the indices was assessed by calculating condence intervals for the differences between groups in mean scores using normal distribution theory and application of the central limit theorem.2 The difference between two groups in mean score divided by the standard deviation (sd) of individual scores was used as a measure of effect size. The intraclass correlation coefcient was used as a measure of testretest reliability.3 Sensitivity and responsiveness to change were assessed by comparing change in mean scores before and after interventions known to be effective in improving voice quality with the sd of baseline scores and the sd of change scores. The analyses of testretest reliability and responsiveness to change were restricted to subjects for whom there were no missing data.

Results
One main issue is the proportion of voices for which the Computer Speech Laboratory (CSL) acoustic analysis is even possible. The number of subjects for whom an assessment was attempted and the number of subjects who

Non-intervention cohort 36 53.0 (2088) 7 (19.4) 29 (80.6) 11 (30.6) 17 (47.2) 2 (5.6) 6 (16.7) 0

All 181 52.7 (1788) 54 (29.8) 127 (70.2) 57 74 25 24 1 (31.7) (41.4) (13.9) (13.3)

Subjects with normal voice 50 41.4 (2376) 19 (38.0) 31 (62.0)

All subjects 231 50.2 (1788) 73 (31.6) 158 (68.4)

90 (62.1) 55 (37.9)

2004 Blackwell Publishing Ltd, Clinical Otolaryngology, 29, 538544

540 P.N. Carding et al.

Table 2. Number of assessments attempted, number of subjects able to provide an analysable voice sample, and mean scores by index by assessment by cohort Cohort Subjects with abnormal voice Intervention cohort Total sample size (n in retest study) Initial acoustic assessment Number of subjects [n (%)] Jitter Analysable [n (%)] Mean (SD) Shimmer Analysable [n (%)] Mean (SD) NHR Analysable [n (%)] Mean (SD) Retest assessment Number of subjects [n (%)] Jitter Analysable [n (%)] Mean (SD) Shimmer Analysable [n (%)] Mean (SD) NHR Analysable [n (%)] Mean (SD) Post-treatment assessment Number of subjects [n (%)] Jitter Analysable [n (%)] Mean (SD) Shimmer Analysable [n (%)] Mean (SD) NHR Analysable [n (%)] Mean (SD) 145 (14) 142 (97.9) 110 (77.5) 6.31 (5.13) 110 (77.5) 11.65 (8.08) 105 (76.6) 0.35 (0.35) 14 (100) 13 (92.9) 4.98 (2.85) 13 (92.9) 11.82 (6.98) 13 (92.9) 0.25 (0.18) 141 (97.2) 134 (95.0) 4.10 (3.83) 134 (95.0) 9.07 (9.06) 132 (93.6) 0.24 (0.24) Non-intervention cohort 36 (36) 33 (91.7) 29 (87.9) 5.09 (2.78) 29 (87.9) 12.01 (7.88) 29 (87.9) 0.24 (0.12) 33 (91.7) 30 (90.9) 5.05 (2.79) 30 (90.9) 11.72 (7.91) 30 (90.9) 0.23 (0.10) All 181 (50) 175 (96.7) 139 (79.4) 6.05 (4.75) 139 (79.4) 11.73 (8.01) 134 (78.8) 0.33 (0.32) 47 (94.0) 43 (91.5) 5.03 (2.78) 43 (91.5) 11.75 (7.56) 43 (91.5) 0.23 (0.13) Subjects with normal voice 50 (50) 49 (98.0) 49 (100) 4.43 (2.71) 49 (100) 8.74 (6.24) 49 (100) 0.35 (0.27) 45 (90.0) 45 (100) 4.49 (2.72) 45 (100) 8.42 (5.08) 45 (100) 0.36 (0.28) All subjects 231 (100) 224 (97.0) 188 (83.9) 5.63 (4.36) 188 (83.9) 10.95 (7.68) 183 (83.6) 0.33 (0.31) 92 (92.0) 88 (95.7) 4.75 (2.74) 88 (95.7) 10.05 (6.59) 88 (95.7) 0.27 (0.26)

were able to provide a voice sample that could be analysed is given in Table 2. A total of 231 subjects were recruited to the study, comprising 181 dysphonics and 50 normal people. Approximately 20% dysphonic voices were not analysable at initial acoustic assessment. All normal voices were analysable for all acoustic parameters. Subjects in the intervention cohort were more likely to be able to provide an analysable voice sample after treatment than before (odds ratio 7.3; 95% CI: 2.520.6). Mean and sd scores for the three elements of the acoustic assessment are given. Jitter, shimmer and NHR values at initial assessment can be compared for the intervention and non-intervention cohorts and for subjects with a normal voice. Similarly, acoustic retest values are available for all subject cohorts. Finally, posttreatment acoustic values are presented for the patients who received intervention. (Mean scores for pre- and post-treatment assessments based on completed pairs are given in Table 5.)

distribution of scores The rst three plots in Fig. 1 show the distribution of jitter, shimmer and NHR scores obtained at the initial assessment. The horizontal line towards the centre of each box denotes the median score, the box extends from the 25th percentile (lower edge) to the 75th percentile (upper edge), and thus indicates the interquartile range (IQR). The lines emerging from the boxes (whiskers) give an indication of the range of the data. Outliers (observations which are more than 1.5 IQRs from the edge of the box) are individually plotted (as a circle). Although the scores tend to be positively skewed the distributions suggest that it would be reasonable to consider parametric procedures such as the t-test when analysing these variables. These acoustic indices do not suffer from any obvious oor or ceiling effects.

2004 Blackwell Publishing Ltd, Clinical Otolaryngology, 29, 538544

Change of acoustic measures of voice quality 541

Initial jitter score 30 25 20 15 10 5 0 Intervention Non-intervention Normal voice 20 10 0 50 40 30

Initial shimmer score

Intervention Non-intervention Normal voice

Jitter by cohort
Initial noise to harmonic ratio 2 1.5 1 .5 0

Shimmer by cohort

Intervention Non-intervention Normal voice

NHR by cohort
Pre-intervention 30 25 20 15 10 5 0 SLT Surgery 20 10 0 SLT Surgery Post-intervention 50 40 30 Pre-intervention Post-intervention

Jitter by intervention by time

Pre-intervention 2 1.5 1 .5 0 SLT Surgery Post-intervention

Shimmer by intervention by time

NHR by intervention by time

Figure 1. Distributionof jitter, shimmer and NHR scores by cohort and intervention.

discriminatory power The jitter score discriminates only between intervention subjects and subjects with a normal voice (effect size 0.43; Table 3). It does not discriminate between the non-intervention (dysphonic) cohort and the normal voice (effect size 0.15) cohort and it does not discriminate between the two cohorts of subjects with abnormal voices (effect size 0.28). Both cohorts of subjects with abnormal voices have signicantly higher mean shimmer scores than the cohort of subjects with a healthy voice (effect sizes 0.38 and 0.43). Subjects with an abnormal voice in the non-

intervention cohort had a lower mean NHR than subjects in the other two cohorts. Although these differences were statistically signicant (P < 0.05), the actual effect sizes involved were fairly small (0.36 in both cases); the reduced mean score in the non-intervention cohort is probably just a chance occurrence. reliability The intraclass correlation coefcient was used to assess test retest reliability (Table 4). For subjects with dysphonia, reliability was at best moderate (for jitter and shimmer) and

2004 Blackwell Publishing Ltd, Clinical Otolaryngology, 29, 538544

542 P.N. Carding et al.

Table 3. Pairwise comparison of the three cohorts for each acoustic measure: difference in mean scores (95% CI) Comparison Acoustic measure Jitter Shimmer Noise-to-harmonic ratio Intervention group versus normal subjects 1.88 (0.34 to 3.42) 2.91 (0.34 to 5.48) )0.00 ()0.11 to 0.11) Non-intervention group versus normal subjects 0.65 ()0.63 to 1.93) 3.28 (0.06 to 6.49) )0.11 ()0.22 to )0.00) Intervention subjects versus non-intervention subjects 1.23 ()0.73 to 3.18) )0.36 ()3.68 to 2.96) 0.11 ()0.02 to 0.24)

Table 4. Testretest reliability coefcients (95% CI) Subject group Acoustic measure Jitter Shimmer Noise to harmonic ratio Dysphonic (n 39) 0.46 (0.23 to 0.70) 0.40 (0.18 to 0.67) 0.33 (0.11 to 0.63) Normal voice (n 45) 0.73 (0.58 to 0.85) 0.55 (0.35 to 0.74) 0.68 (0.51 to 0.82)

Discussion
This study aimed to evaluate the discriminatory power, reliability, and the effect size of the sensitivity to change of three commonly used acoustic parameters as measured by the MDVP; jitter, shimmer and NHR. The clinical sample comprised unselected dysphonic outpatients and hence the study was able to determine the applicability of these acoustic measures in a normal clinical setting. Our results report that 20% of dysphonic voices cannot be analysed at all for acoustic measurement. These voices represent type III signals23 which have insufcient periodic structure to allow the software to produce any measurement of cycle-to-cycle perturbation. It is also very likely that the data sample included a large number of type II acoustic voice signals which, according to Titze23 further limits the utility and reliability of perturbation measurement. This point will be discussed later. There was a small to moderate size effect of all three acoustic parameters following effective clinical intervention (as measured by perceptual ratings and voice quality of life measures). The modest period-free reliability will limit the attainment of the best possible change effect size estimate, because change applies to reliable variance and not error variance. It is possible that we could have achieved larger size effects with sub-groups of data. For example, previous authors have suggested a close correlation between specic acoustic

poor for NHR. Reliability of the measures is better when used with subjects with a normal voice but this is the sample to whom the measures are least relevant. responsiveness to change The lower set of box plots in Fig. 1 give a visual indication of the change in each of the acoustic indices following each of the interventions. For each index, for each intervention, there was a reduction in scores following the intervention (Table 5). The changes in mean scores represent effect sizes that can be considered as small to moderate. The effect sizes are similar mostly around one-third of a sd for both interventions but the difference in mean scores reaches statistical signicance only for SLT (most likely because of the larger sample size).

Table 5. Change in acoustic indices following intervention Mean score (sd) Intervention SLT Surgery Index Jitter Shimmer HNR Jitter Shimmer HNR n 63 63 58 39 39 38 Pre-treatment 6.1 12.3 0.34 6.1 9.5 0.32 (4.4) (8.3) (0.32) (6.1) (6.5) (0.34) Post-treatment 4.1 9.5 0.23 4.2 7.7 0.23 (4.1) (9.4) (0.28) (4.2) (8.4) (0.18) Difference (95% CI) )2.1 ()3.5 to )0.6)** )2.8 ()5.5 to )0.03)* )0.10 ()0.21 to 0.002) )1.9 ()4.3 to 0.4) )1.8 ()5.1 to 1.4) )0.11 ()0.24 to 0.01) Effect size ES1 0.47 0.34 0.32 0.32 0.28 0.34 ES2 0.36 0.25 0.26 0.27 0.18 0.31

n number of subjects with a score both pre- and post-treatment. ES1 change in mean score divided by sd of baseline scores (Cohens D). ES2 change in means score divided by sd of change scores. *P < 0.05; **P < 0.01. 2004 Blackwell Publishing Ltd, Clinical Otolaryngology, 29, 538544

Change of acoustic measures of voice quality 543

parameters and certain perceptual voice quality types (e.g. jitter and roughness).20 However, these relationships remain unclear and recent work22 suggests that the mapping of acoustic features to different voice qualities is highly complex and multidimensional. Furthermore, previous work by our group, using the same data set suggests limited reliability of perceptual ratings of voice quality.25 Therefore, correlation analysis between the two independent measures of voice quality may only serve to further compound the matter. Reliability of the testretest data were poor for the clinical population and moderate for the normal voice samples. We recognized that dysphonic voices are inherently variable and therefore attempted to minimize this effect by allowing subjects to practice prior to the recording sample and to perform the repeated measures within 2 h. It is possible that recording multiple voice samples and averaging the results (as suggested by Titze23) may improve the reliability scores and this could be the subject of a further study. Furthermore, there are many sources of potential variability within the process of acoustic analysis. Previous studies have examined the intraand inter-system reliability of a number of computerized acoustic analysis programmes and found them to be only moderately reliable.1820 This reliability would appear to be best in near-to-normal (type I) voice signals but decrease further as the voices become more severely dysphonic (less periodic). Within-subject (repeated measures) analysis has also been found to be only moderately reliable even in nondysphonic control subjects.19 From the analysis of the reliability of our data, it is not possible to recommend acoustic analysis as an independent measure of voice outcome. This poor reliability may partly explain the large sd of these acoustic indices which will, in turn, affect the measurements of sensitivity to change. The mean scores for all three acoustic measures decreased following intervention. The actual change was signicant only for jitter and shimmer scores following voice therapy but statistical signicance depends upon the sample size and there were more subjects who received voice therapy than surgery. For each intervention and for each acoustic measure, the effect size was approximately one-third of sd. This is a fairly small change in populations where there is reason to expect considerable improvements in voice quality. One possible reason for the small effect size is that, because the measures are not particularly reliable, there is a lot of inherent variability even within subjects resulting in large sd for each measure in comparison with mean scores. The evidence from this study suggests that these acoustic measures have limited sensitivity to change when used indeterminately on all voices. The sensitivity-to-change data illustrate the danger of routinely applying isolated acoustic measurement values to measure change in voice quality following intervention. Furthermore, given the likelihood

that the clinical voice sample contains a large number of type II acoustic voice signals23 (i.e. signals that contain intermittancy and aperiodic segments) the routine applicability of acoustic analysis should be further questioned. Titze23 provides guidelines to determine the appropriateness of voice samples for acoustic analysis. These include visual inspection of the signal to ensure sufcient signal periodicity, exclusion of atypical characteristics in the selected sample and selection of the most stable segment. Our study would suggest that these selection criteria would render a large proportion of our clinical voice sample as unsuitable for acoustic analysis. It is well recognized that automated acoustic analysis systems may provide perturbation measures even when the underlying signal is too noisy to provide reliable period detection.24,26 This acoustic analysis data are totally erroneous. It is important to note that, in our study, a number of patients demonstrated clinically signicant improvement following intervention but could not be measured using acoustic analysis methods. A total of 32 patients had type III voice signals prior to treatment and could not therefore be analysed. However, 91% of these patients had analysable voice signals post-treatment but could not be included in these data for obvious reasons. This has additional implications in the potential employment of acoustic analysis measurement as a clinical measure of intervention outcome. Finally, assessment of sensitivity to change of these acoustic measures is difcult to do without a gold standard against which to make comparisons. We are currently preparing publication of further data that allows comparison of acoustic, perceptual and patient self-report voice outcome measurements. One main aim of this report will be to discuss the relative sensitivity of each measure in comparison with each other. In conclusion, the acoustic analysis measures used in this study did not demonstrate adequate reliability, effect size or sensitivity to change to recommend their routine use to measure outcome in clinical trials of interventions aimed at improving voice quality. Our results suggest that perturbation measures may have specic value with particular voice quality types; however, this relationship requires further investigation. It is possible that the reliability of acoustic measurement of the voice may be enhanced by adopting strict signal selection criteria and multiple sampling analysis. However, this practice is against clinical practicality and routine use. Voice outcome measurement should reect the multidimensional aspects of voice quality27 and techniques should be selected specically to elucidate and support other clinical measures. Acoustic measurements of the speech signal are perhaps best used as evidence to support clinical impression and, where appropriate, as part of an integrated and complementary clinical evaluation.26

2004 Blackwell Publishing Ltd, Clinical Otolaryngology, 29, 538544

544 P.N. Carding et al.

Acknowledgements
This study was funded by a Health Services Research Grant from the Wellcome Trust.

References
1 Bertino G., Bellomom A., Ferrero F.E. et al. (2001) Acoustic analysis of voice quality with or without false vocal fold displacement after cordectomy. J. Voice 15, 131140 2 Bland M. (2000) An Introduction to Medical Statistics, 3rd edn. Oxford University Press, London 3 Dunn G. (1989) Design and Analysis of Reliability Studies: the Statistical Evaluation of Measurement Errors. University Press, London 4 Reijonen P., Lehikoinen-Soderlund S. & Rihkanen H. (2002) Results of fascial augmentation in unilateral vocal fold paralysis. Ann. Otol. Rhinol. Laryngol. 11, 256259 5 Peretti G., Provenzano L., Piazza C. et al. (2001) Functional results after type I thyroplasty with the Montgomerys prosthesis. Acta Otorhinolaryngolologica Italica 21, 156162 6 Aluffi P., Policarpo M., Cherovac C. et al. (2001) Postthyroidectomy superior laryngeal nerve injury. Eur. Arch. OtoRhino-Laryngol. 258, 451454 7 Uloza V. (1999) Effects of voice by endolaryngeal microsurgery. Eur. Arch. Otol. Rhinol. Laryngol. 256, 312315 8 Roy N., Ryker K.S. & Bless D.M. (2000) Vocal violence in actors: an investigation into its acoustic consequences and the effects of hygienic laryngeal release training. J. Voice 14, 215230 9 Holmberg E.B., Hillman R.E., Hammarberg B. et al. (2001) Efcacy of a behavioural based voice therapy protocol for vocal nodules. J. Voice 15, 395412 10 Rovirosa A., Martinez-Celdran E., Ortega A. et al. (2000) Acoustic analysis after radiotherapy in T1 vocal cord carcinoma: a new approach to the analysis of voice quality. Int. J. Rad. Oncol. Bio. Phys. 47, 7379 11 Aref A., Dworkin J., Devi S. et al. (1997) Objective evaluation of the quality of voice following radiation therapy for T1 glottic cancer. Rad. Oncol. 45, 149153 12 Hamdan A.L., Sharara A.I., Younes A. et al. (2001) Effects of aggressive therapy on laryngeal symptoms and voice characteristics in patients with gastro-esophageal reux. Acta OtoLaryngol. 121, 868872 13 Balter M.S., Adams S.G. & Chapman K.R. (2001) Inhaled beclomethasone dipropionate improves acoustic measures of voice in patients with asthma. Chest 120, 18291843

14 Sanabria J., Ruiz P.G., Gutierrez R. et al. (2001) The effect of levodopa on vocal function in Parkinsons disease. Clin. Neuropharmacol. 24, 99102 15 Hadjitodorov S. & Mitev P. (2002) A computer system for acoustic analysis of pathological voices and laryngeal disease screening. Med. Eng. Phys. 24, 419429 16 Boyanov B. & Hadjitodorov S. (1997) Acoustic analysis of pathological voices. A voice analysis system for the screening of laryngeal diseases. Eng. Med. Biol. 16, 7482 17 Niedzielska G. (2001) Acoustic analysis in the diagnosis of voice disorders in children. Int. J. Ped. OtoRhinoLaryngol. 57, 189193 18 Sapienza C.M., Walton S. & Murray T. (2000) Adductor spasmodic dysphonia and muscular tension dysphonia; acoustic analysis of sustained phonation and reading. J. Voice 14, 502 520 19 Dixon H.S. (1999) Dysphonia and delayed food allergy: a provocation/neutralization study with videolarngostroboscopy. Otol. Head Neck Surg. 121, 418429 20 Gonzalez J., Cervera T. & Miralles J.L. (2002) Acoustic voice analysis: reliability of a set of multidimensional parameters. Acta Otorhinolaringologica Espanola 53, 256268 21 Bough D., Reinhardt J.H., Sataloff R.T. et al. (1996) Intrasubject variability of objective voice measures. J. Voice 10, 166174 22 Rabinov R.C., Kreiman J., Gerratt B.R. et al. (1995) Comparing reliability of perceptual ratings of roughness and acoustic measures of jitter. J. Speech Hear Res. 38, 2632 23 Titze I. (1995) Workshop on Acoustic Voice Analysis; Summary Statement. National Centre for Voice and Speech, The University of Iowa, IA 24 Frohilch M., Mihcaelis D., Strube H.W. et al. (2000) Acoustic voice analysis by means of the hoarseness diagram. J. Speech Hear Res. 43, 706720 25 Webb A., Carding P., Deary I. et al. The reliability of three perceptual evaluation rating scales for dysphonia. Eur. Arch. OtoRhinoLaryngol. (in press) 26 Hillman R.E., Montgomery W.W. & Zietels S.M. (1997) Appropriate use of objective measures of vocal function in the multidisciplinary management of voice disorders. Curr. Opin. Otolaryngol. Head Neck Surg. 5, 172175 27 Carding P.N. (2000) Evaluating Voice Therapy; Measuring the Effectiveness of Treatment. Whurr Publishers, London

2004 Blackwell Publishing Ltd, Clinical Otolaryngology, 29, 538544