Food Quality and Preference 18 (2007) 425439

www.elsevier.com/locate/foodqual

Food quality certication: An approach for the development

of accredited sensory evaluation methods q
F.J. Perez Elortondo

a,*

, M. Ojeda a, M. Albisu a, J. Salmeron a, I. Etayo a, M. Molina

Laboratorio de Analisis Sensorial, Facultad de Farmacia, Universidad del Pas Vasco/Euskal Herriko Unibertsitatea,
Paseo de la Universidad, 7, 01006 Vitoria, Gasteiz (Alava), Spain
Consejo Regulador de la Denominacion de Origen Queso Idiazabal, Granja Modelo de Arkaute, Apartado 46, Arkaute (Alava), Spain
Available online 20 May 2006

Abstract
The standardization and accreditation of sensory quality evaluation methods is a pressing need for the certication of food products,
particularly for foods and beverages with specic sensory characteristics, such as those with a protected designation of origin (PDO).
This study describes a proposal for the development of accredited sensory evaluation methods, showing as an example an application
from a specic certied product (PDO Idiazabal cheese). A training and qualication process for expert panelists is required. In cheese,
panelists score quality of overall sensory parameters (shape, rind, paste colour, eyes, odour, texture, avour and aftertaste) on a scale,
based on how close the product lies to a specic quality standard. Panelists justify the quality scores given on the basis of the absence/
presence of specic characteristics in the product and/or the presence of defects. Training requires the prior establishment of references
for both characteristics and defects. Qualication trials determine whether or not the expert panelists (both individually and as a panel)
are appropriately qualied to carry out the sensory evaluation. This work also shows the quality control maintenance of qualications
for the expert panellist. This approach could be generalized to any type of food and beverage as a reference for the accreditation of sensory quality evaluation methods according to ISO 17025. In this way, each product manufacturer would be able to dene its quality
standard and, on the basis of this standard, carry out the sensory evaluation using a panel specically trained for this purpose.
 2006 Elsevier Ltd. All rights reserved.
Keywords: Sensory quality evaluation; ISO 17025 sensory method accreditation; Food quality certication; Protected designation of origin (PDO);
Idiazabal cheese

1. Introduction
The scientic strategy most acceptable for sensory quality evaluation takes into account the relation from two
types of data: tests with consumers (aective class, hedonic
type test) and trained analytical panels (descriptive class,
analytic type test). The relation between the two makes
it possible to determine sensory proles best adapted to
the concept of the product quality in the target market,
q

A poster of this paper was accepted and presented during the

European Conference on Sensory Science of Food and Beverages A
sense of identity in Florence (Italy) from 26th to 29th September 2004.
*
Corresponding author. Tel.: +34 945 013075; fax: +34 945 013014.
E-mail address: franciscojose.perez@ehu.es (F.J. Perez Elortondo).
0950-3293/$ - see front matter  2006 Elsevier Ltd. All rights reserved.
doi:10.1016/j.foodqual.2006.05.002

enabling large companies to establish control activities,

improve quality, and develop new products. This strategy,
however, has limitations with respect to the possibility of it
being applied by small producers with typical sensory characteristic products, such as those with protected designation of origin (PDO).
The integration of the two types of information requires
high level technical work, where multidisciplinary research
is needed (psychology, marketing, dynamic group techniques, sensory methodology, statistical multivariate
analysis).
There is a standardized method for the selection, basic
training and development of sensory proles, but a lack
of standards for consumer tests, this being one of the main
needs of sensory analysis standardization.

426

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

In laboratories that only use descriptive techniques, typically the number of positive descriptors is limited (key
attributes) and it is doubtful that it is even possible to
achieve a product more in line with a given sensory prole,
but where quality is insucient due to the presence of a
defect.
Finally there are some products, which, although they
should be market orientated to ensure successful sales, their
sensory attributes are inuenced by certain origins or processing factors. This is the case of PDO products that need
to gain the acceptance of consumers while at the same time
respecting their sensory typicalness. In many cases, these
producers are small enterprises, where it is economically
impossible for them to develop and introduce systematic
quality control on the end product from hedonic and analytical information.
Questions arise such as the followings: Apart from the
opinion of the consumer, should the inuence of technical,
cultural, social and environmental factors be taken into
account to obtain dierentiated products? Would it be
appropriate to propose a sensory analysis method validated and recognised (accredited), allowing a direct evaluation of the quality previously dened for a product, by
means of a panel of expert panelists trained specically
for the purpose? Using this method, should we take into
account the possible defects known in the product that
are important for its identity? How do we integrate said
possible defects in the quality evaluation?
Generally, in quality certication of PDO products,
regulations underline the fact that these products should
present sensory characteristics in accordance with the
judgement of a Sensory Committee. These panelists,
who may be labelled expert judges of a specic product,

Quality definition of the product

Specific development of the method

should evaluate not only the presence of certain specic

sensory characteristics in the product, but also the absence
of possible defects. The members of these panels are people
with a great deal of experience in tasting a specic product,
but in many cases, neither use standardised methods nor
have they been given systematic training. Many of the
PDO products still lack a Sensory Committee, and this is
largely due to the lack of references published on how to
conduct such a task.
Sensory evaluation comprises a set of techniques for
accurate measurements of human responses to foods and
minimizes the potentially biasing eects of brand identity
(Lawless & Heymann, 1998). Sensory characteristics comprising appearance, odour, avour and texture are included
within the quality of food products. There are a limited
number of internationally accepted standards on general
methods in sensory analysis, such as: general guidance
(ISO 6658, 1985), assessors selection and panel training
procedures (ISO 8586, 1993) and sensory tests (ISO 5495,
1983; ISO 4120, 1983; ISO 6564, 1985; ISO 10399, 1991;
ISO 1036, 1994). These standards permit the selection,
basic training of panelists and general application of discriminative and descriptive sensory methods. Using these
standards it is possible to detect and describe sensory dierences between food products but not to establish and evaluate their quality or dene training procedures for experts
in specic foods.
Accreditation means the demonstration of technical
competence. Sensory laboratory accreditation would have
a future and perhaps could be realized in the next two decades (Moskowitz, Munoz, & Gacula, 2003). ISO 17025
(1999) was developed in order to obtain an accepted system
for comparable evaluation of testing activities. According

Selection and basic training of panelists

Specific training of panelists

Individual qualification of the expert panelists

Validation of the method (proficiency tests included)

Quality control maintenance of expert panelists and of validated method

General quality management activities

Fig. 1. Main activities for the accreditation of a sensory quality evaluation method on a specic food product.

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

to the European Community regulations for PDO products, the certifying organizations of these food products
have to comply with the European Standard EN45011
(DOCE, 1992). Consulting the mentioned standard, it
becomes obvious that the analysis for certifying these products, sensory included, should be carried out in laboratories
according to the standard ISO 17025. One of the most
important documents nowadays for standardization of sensory activity is the European publication reference EA-4/09
Accreditation for sensory testing (ENAC, 2003) that
supplements ISO 17025 and provides specic guidance on
the accreditation of sensory laboratories. This document
gives detailed guidance for the interpretation of ISO
17025 for those undertaking sensory examinations. Accreditation bodies will only accredit laboratories for sensory
methods that have been fully documented and validated.
Some laboratories have accredited general discriminative,
descriptive and consumer sensory tests, but there is no previous experience in the accreditation of sensory quality
evaluation methods of specic products.
The aim of this study is to describe, for example, a sensory evaluation method that serves as a basis for the sensory quality certication of a specic product (PDO
Idiazabal cheese), as well as the training and qualication
process for expert panelists required for implementation.
The paper presents the experience of four years of work
and shows a general perspective across all the activities
(Fig. 1) developed in an accredited sensory laboratory
(June, 2005) according to ISO 17025:http://www.enac.es/
html/anx/le/UPV_EHU_LE1020_REV1.pdf.
2. Material and methods
2.1. Expert panelists for sensory quality evaluation
of a specic product
ISO standards on the selection and basic training of
panelists may be used as a reference during the initial training stages of experts, although such standards do not establish the required level needed to pass these preliminary
tests. Afterwards, expert judges should be trained specically on the sensory quality of the product under study
(characteristics and defects) and for the specic application
of the evaluation method.
2.2. Installations, equipment, chemical and foodstus
specic references
An accredited sensory analysis laboratory (SAL) should
have a meeting room according to the ISO standard, standardized booths and an isolated sample preparing area.
The other equipment in a SAL is not sophisticated, in order
to comply with the accreditation standards. This equipment is basically limited to thermo hygrometers and continuous read-out thermometers, refrigerators, tempered
sample chambers, weighing scales, distilling equipment
and/or water deionisation and volumetric material, chemi-

427

cal compounds and foodstus used for the standardisation

of references.
The harmonized use of common references by dierent
laboratories represents in itself a beginning of standardization and currently it could be valid in the accreditation of
specic sensory evaluation methods. However, as pointed
out by Issanchou, Schlich, and Lesschaeve (1997) each
product needs its own adequate references which can be
completely dierent from one type of cheese to another,
thus making it impossible to generalize a set of standards
for every variety.
2.3. Specic samples of the product
During the specic training of panelists, a wide variety
of product samples is essential in order to accommodate
the highest possible quality variability in the product.
On a daily basis, the systematic evaluations make it possible for the laboratory to identify interesting product samples for dierent purposes, such as harmonize and improve
the consensus among the panel members, introduce quality
control tests, and increase the experience of the panelists
for the identication of defects.
2.4. Data treatment
A coding and internal recording system should ensure
traceability of the results and all the information from
the activities of the SAL.
Data treatment is applied at very dierent levels: qualication and follow-up of the panelists performance,
method validation, quality control tests, prociency testing
schemes, treatment of the results and test reports. Generally, statistical descriptive analysis and average comparison
tests are appropriate; more sophisticated multivariate analyses are not essential.
3. Results and discussion
3.1. Sensory quality denition of the product
Sensory analysis is a research discipline that should
become one of the main driving forces to ensure the quality
and protection of PDO food products (Bertozzi & Panari,
1993; Perez Elortondo, Barcenas, Casas, Salmeron, &
Albisu, 1999). To establish the sensory quality of a PDO
product and dene a method for its evaluation, it is necessary to have in-depth knowledge of the sector involved
(type of company, production systems, and product variability) and to compile scientic and market information
related to the type of product under study. A working
group on sensory analysis should be set up to reach a consensus of opinion on progress, both in terms of sensory
quality denition and the evaluation method. The members
of this group are a decisive element for ensuring the success
of the work. It is very important to be able to rely on people with acknowledged prestige and experience in the sector

428

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

and that have something to say. Members of this working

group should include product processors, restaurateurs,
and technicians in sensory analysis, gastronomic critics
and consumers.
In order to achieve an initial sensory quality denition
draft and a consensus over the evaluation method, experience has taught us that several meetings are needed, lasting
up to 3 h and held weekly over a period of approximately
612 months, depending on the complexity of the product
and on the number of companies involved. These meetings
should include sensory evaluation sessions to discuss the
sensory variability of the product, to generate and select
sensory descriptors, and to record any possible defects.
At this stage, it is important to identify the essential sensory attributes linked to the origin and/or particular systems of production. It is advisable to corroborate the
identication of these essential attributes by means of a
parallel research procedure, using descriptive methods by
an analytical trained panel. These key attributes distinguish
the PDO product and ensure its typicalness and protection
in terms of possible imitations.
The result of this work (sensory quality denition standard) should be conveyed to the production sector for
acceptance as a sensory reference for the control and
improvement of the product quality. Follow-up of adjustment to this initial sensory quality standard should continue, because changes may occur in the consumer
preferences, in the production systems and/or new knowledge in research sensory characterisation may also be
available.
As an example this procedure made it possible to establish the standard sensory quality denition of a PDO product such as Idiazabal cheese (Table 1) a decade ago.
Nowadays this denition is the basic reference which is
used for applying the accredited sensory evaluation method
developed by our SAL.
3.2. Selection and basic training of panelists
3.2.1. Selection of panelists
SAL comprises three selection stages: recruiting, selection and preliminary formation. SAL includes two ways
of recruiting: internal (when candidates belong to the University), or external (when candidates are expert technicians, restaurateurs or consumers). Initial selection of
new judges is carried out by questionnaire, to determine
personal aspects, such as the aptitude for foods, communication, health, availability and other factors of a general
nature.
Preliminary formation trains new judges by familiarizing them with the discipline of sensory analysis. Explanations are given based on theory, and selection tests are
conducted. Test results, used as a basis for selecting panelists, address the following objectives: to detect inabilities,
determine sensory sensibility and evaluate the potential of
the candidates in describing and communicating their
perceptions.

Table 1
Sensory standard denition of Idiazabal cheese used in quality evaluation
(adapted from Perez Elortondo, 1993)
Variable

Optimal situation (reference)

Shape

Cylindrical, well-proportioned, 812 cm high, 1030 cm

diameter and 0.93 kg weight. Markedly at faces.
Slightly convex sides. Rounded edges on small cheeses,
sharp edges on larger ones
Hard. Smooth, without trace of outside agents. Slight
signs of the cloths used. Homogenous colour, from a
pale yellow or whitish grey to a darker grayish-brown
in the case of smoked cheeses. Absence or light marks
from the trays on one of the faces
Homogeneous. Variable (from ivory to straw yellow).
Matt. Narrow and slightly darker border
Random distribution. Absence or not very numerous.
Mostly of irregular shape. Smaller than a grain of rice.
Absence of cracks
Intense odour, ewes milk, clean and sharp. Slightly
pungent. Variable acidity (intensity null to medium) and
sweetness (intensity null to medium). Medium intensity
smokiness with smoked cheeses
Slight elasticity. Medium rmness. Weak granularity
Characteristic avour, balanced and intense: character
of mature ewes milk, somewhat of natural rennet
avour, clean and consistent. Slightly pungent. Variable
sweetness and acidity. Absence of bitterness. Medium
saltiness. Mild to medium smokiness in smoked cheeses
Continuity with respect to the characteristic avour.
Persistent. Pronounced

Rind

Paste colour
Eyes

Odour

Texture
Flavour

Aftertaste

Table 2 shows the tests and selection criteria for new

panelists. The type of test and concentrations of the references are based on ISO standards, although some selection
criteria are internal to our SAL due to the absence of specications in the standards. Selection tests are of a basic and
general nature, regardless of the type of product to be
tested by the panelists. Each test is conducted twice.
SAL establishes that the judges achieving a P75% success rate out of the total number of tests conducted are
to be accepted, and will proceed on to the following basic
training stage. Those obtaining a success rate between
60% and 75% may repeat once the tests failed. If after this
test they achieve a 75% success rate, then they are to be
accepted; lower success rates mean rejection from participating in the subsequent basic training.
3.2.2. Basic training of panelists
Table 3 shows the tests and criteria required for the panelists to pass the basic training tests, regardless of the type
of food being tested. The purpose of this training is twofold. First, the training provides judges with the basic elements of the standardized techniques used in sensory
analysis, developing their aptitude to detect, recognize
and describe the sensory stimuli involved. Second, the
training allows judges to become competent in applying
these basic techniques to any food item. The training programme encompasses both theory and practice. As with the
selection tests, the type of basic training test and the concentrations of the references are based on ISO standards,

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

429

Table 2
Tests and criteria for selection of panelists

Table 3
Tests and criteria for basic training of panelists

1. Viewing colours test (Ishihara test): If the candidate views colour

normally, he/she will attain certain numbers. Full points are required

1. Training tests in detecting and recognizing smell and sapid substances

1A. Two duo-trio tests:
(a) Samples representing odours, at concentrations higher than the
recognition threshold. A sample of each type is presented to each
candidate, and they are left to familiarize themselves with the same.
Then they are presented, at random, with two series of the same
samples. Candidates should pair them o with the original samples. A
minimum 80% success rate is required in the pairing o test
(b) Four sets of three sapid samples with the combinations AAB, ABA,
BAB, BBA are presented. Samples A and B correspond to two dierent
branches of the same product. They are asked to examine the samples in
each set from left to right so that the reference sample (the rst on the
left) is examined rst, and to indicate which of the other two samples is
identical to the reference. A minimum 75% correct answer will be
required, both in the sapid and in the smell tests
1B. Pairs comparison test. Panelists are presented with two pairs of
samples for each characteristic to be evaluated in terms of avour and
odour. Panelists must compare, within each pair of samples, which of
them presents a higher degree of intensity in terms of a particular
attribute (forced judgement). A minimum 75% correct answer will be
required, both in the sapid and in the smell tests

2. Colour recognition test. Ranking test: Candidates are presented with

ve random colour strips from the Yolk Colour Fan for colours of egg
yolk. They are asked to put the strips in ascending order in terms of the
degree of colour intensity. A maximum inversion is required in adjacent
pairs
3. Taste sensitivity test (taste identication). 13 dissolutions are presented
to judges. Each judge will taste, successively, the content of each
dissolution, following the order in which they are presented, without
going back to the samples previously tasted. A minimum success rate of
80% is required
4. Sapid substance recognising tests
4A. Duo-trio test: Samples representing basic tastes, at concentrations
higher than the recognition threshold. A sample of each type is
presented to each candidate, and they are left to familiarize themselves
with the same. Then they are presented, at random, with two series of
the same samples. Candidates should pair them o with the original
samples. A minimum 80% success rate is required in the pairing o test
4B. Triangular test: The concentration of the product being tested
should be at a supraliminal level. The candidate should detect which
sample is dierent. Four trials will be presented. A minimum 75%
success rate is required
4C. Ranking test: Panelist are asked to classify samples in order of
growing intensity. The order of presentation will be the same for all
candidates. A maximum inversion is required in adjacent pairs
5. Olfactory substance recognition tests
5A. Ranking test: Panelists are asked to classify samples in order of
growing intensity. The order of presentation will be the same for all
candidates. A maximum inversion is required in adjacent pairs
5B. Odour description test: Samples containing dierent smelling
substances, at concentrations higher than the recognition threshold.
Candidates must mentally classify the sensory perceptions identied and
note down the family of odours and/or descriptors. 3 points
(identication or correct description); 2 points (description in general
terms); 1 point (identication or description of an appropriate association
following discussion); 0 for no answer. Candidates must score at least 65%
6. Texture recognition tests
6A. Ranking test: Solid samples (foods). Panelists are asked to order
the samples in ascending order, in terms of elasticity and granularity. A
maximum inversion is required in adjacent pairs
6B. Test for describing texture: Solid samples are presented. Panelists are
asked to describe their properties as regards texture, noting down the
term and/or terms most frequently linked with its texture. 3 points
(identication or correct description); 2 points (description in general
terms); 1 point (identication or description of an appropriate association
following discussion); 0 for no answer. Candidates must score at least 65%

although some selection criteria are internal to our SAL in

view of the absence of specications in the standards.
During the basic training stage, the panel leader carries
out a follow-up evaluation to assess the interest and dedication of each panelist and on their sensory capacities
and correct application of the sensory methodology,
recording these in the laboratory record. The judges who
have a success rate of at least 75% out of the total number
of tests in duplicate are considered to have passed the basic
training stage. Those who do not achieve this success rate
may repeat once the tests not passed. Finally, the labora-

2. Training in the use of scales

2A. Use of one-dimensional descriptors: Olfactory and sapid samples
are presented at dierent concentrations. To enable the panelists to
become acquainted with the concepts of classication, with the help of
dierent scales (structured continuous and non-structured scales,
categories or discontinuous scales), they will order the odour and
avour stimuli in terms of the intensity of each particular attribute.
They will be asked not to invert the order of the samples in more than one
of the characteristics evaluated, considering both the odour and avour
scales separately
2B. Use of multidimensional descriptors. Odour, avour and texture
samples are presented, at dierent concentrations. Using the same
scales as in the previous sessions, panelists will put the stimuli from the
samples of odour, avour and texture in order, in terms of the intensity
of each particular attribute. They must not invert the order of the samples
by more than one of the characteristics evaluated, considering odour,
avour and texture separately
3. Training in developing and using descriptors
The purpose is to make the panelists aware of the concept of prole. In
order to develop the vocabulary, 2 brands of a food are evaluated. By
means of the Interlinked Kelly Method, panellists will describe the
dierences and similarities between the samples. Once the test has
concluded and following an open discussion between the panel and the
panel leader, the terms which the group considers appropriate for the
sensory description of the products are chosen. Hedonic and intensity
terms will be removed. Synonyms or terms presenting similar meanings
are to be grouped together
3A. Sensory evaluation
Panelists will evaluate the food under study in the previous test, noting
points for intensity, using the attributes nally agreed on. The
evaluation method for the descriptors is established and references will
be used at some point on the scale. Three repetitions per brand is
presented to each panelist. The panel leader will establish the prole for
each product, using the results to illustrate the value of the descriptive
analysis. These proles will be shown to the panelists so that they can
become acquainted with their use
So that the panel as a whole becomes acquainted with discriminating
parameters, a comparison of averages is applied (t-Student test) on each
attribute. In order to verify the discriminating capacity of each member of
the panel, a t-Student test will be applied to the results from each panelist
for each discriminating parameter. An individual discriminating capacity
will be required of at least 50% of the discriminating parameters
considered by the panel as a whole

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

430

tory management personnel decides, which of the candidates continue in future tests, based upon interest, availability and training results.
SAL considers these judges (food panelists) to be qualied to conduct simple standardized sensory analysis, basically discriminating tests, and to participate in the
development of descriptive sensory methods for specic
foods.
3.3. Specic training of panelists: PDO Idiazabal cheese
as example
So that a panelist can evaluate the sensory quality of
Idiazabal cheese, he must be given specic training and

Table 4
List of defects in appearancea
Shape

Rind

Paste colour

Holes

Blowing
Low
Convex
Sunken
Inclined

Colouring
Not developed
Cracks
Extraneous agents
Marks
Mouldy
Dirty

Excess of rind halo

White
Dark
Irregular
Internal moulds
Crystals

Caverns
Large
Cracks
Badly distributed
Numerous
Rounded

References: catalogue of photographs.

attain the qualication of expert judge in a specic product

(Idiazabal cheese).

Table 5
Denition and references for characteristics of odour, avour and aftertaste
Characteristic
Odour
Sheeps milk
Sharp
Pungent
Acid
Sweet
Smoky (only smoked
cheeses)
Flavour
Sheeps milkd
Natural rennetd
Pungent
Sweet
Acid

Bitter (absence)
Salty
Smoky (smoked
cheeses)d
Aftertaste
Continuity

Persistent

Notable
a

Denition

Reference

Smelly sensation frequently associated with acidied

sheeps milk
Sensation of penetration in the nasal cavity
Sensation perceived inside the nasal cavity in the
form of irritation, burning or stinging
Sensory property of pure or mixed substances, the
olfactation which is reminiscent of an acidic sensation
Sensory property of pure or mixed substances, the
olfactation which is reminiscent of a sweet sensation
Odour produced following combustion of wood

15 mL acidied raw sheeps milk 36 h at room temperaturec

Olfactive-taste sensation frequently associated with

acidied sheeps milk
Combination of typical aromas of traditional rennet
Sensation perceived in the mouth as an irritation,
burning or stinging
Elemental taste perceived in the mouth with watery
solutions of diverse solutions such as sucrose or fructose
Elemental taste perceived in the mouth with watery
solutions of diverse organic substances such as citric
or lactic acid
Elemental taste caused by diluted watery solutions of
diverse solutions such as quinine or caeine
Elemental taste caused by diluted watery solutions of
diverse solutions such as sodium chloride
Sensation produced following the combustion of wood

Olfactive-taste sensation maintained after removing

the product that does not dier from the sensations
perceived when it was in the mouth
Olfactive-taste sensation very close to the sensation
perceived when the sample of cheese was in the
mouth, remaining localized a certain time
Olfactive-taste sensation with high intensity perceived
after removing the sample of cheese from the mouth

Add 5 mL H2Oa to 40 lL propionic acid

Dissolve 80 lL isobutyric acid to 10 mL with H2Ob
Dissolve 50 lL acetic acid to 10 mL with H2Ob
Dissolve 0.003 g maltol to 10 mL with H2Ob
Dissolve 5 lL commercial phenolic type liquid smoke aroma to
10 mL with H2Ob
15 mL acidied raw sheeps milk 36 h at room temperaturec
Add 50 mL H2O a 0.5 g rennet paste. Stir for 15 min and lterc
Add boiling 100 mL H2O to 0.5 g cayenne pepper seeds.
Leave for 5 min and lter. 1 mL supernatant is added to 100 g curd
To 1.5 g D(+) sucrose are added 10 mL H2O. Add 6 mL BS to
100 g curd
To 75 g curd 25 g of natural yoghurt are added

Not applicable
To 2.35 g anhydrous sodium chloride add 50 mL H2O. Add 2.4 mL
BS to 100 g curd
Dissolve 5 lL commercial phenolic type liquid smoke aroma to
10 mL with H2Ob
Not applicable

Not applicable

Not applicable

5 mL of base solution (BS).

10 mL of base solution (BS).
c
15 mL of base solution (BS).
d
References of acidied raw sheeps milk avour, natural rennet and smoky, may not be ingested. They are prepared and presented as odour
references.
b

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

3.3.1. Training for appearance

The panel leader explains the sensory denition and the
evaluation method of each of the characteristics (Table 1)

431

and defects (Table 4) for the four appearance parameters

(shape, rind, paste colour, and eyes). Theoretical training is
backed by a specic catalogue of photographs (references)

Table 6
Denition and references of texture characteristics
Characteristic

Denition

References

Elasticity

Aptitude of a cheese sample to rapidly regain its initial

thickness after compression and deformation

Firmness

Resistance of a cheese sample to a very slight

opening and shutting of the jaws

Granularity

Perception at the end of the sensation of the dimensions

and shape of particles in the sample. E.g., rounded grains
(more or less hard when bitten into)

Null: soft butter/raw carrot

Medium: stued olive
High: frankfurter sausage
Null: melted cheese
Medium: frankfurter sausage
High: cooked carrot (5 min)
Null (oury): potato puree
Medium (granular): small couscous grain
High (coarse): medium couscous grain

Table 7
Group of defects and references for odour, avour and aftertaste
Defects

Denition

Reference

Vinegar

Smelly sensation associated with white wine vinegar

Acid (in excess)

Organoleptic sensation of pure or mixed substances

with an olfactation reminiscent of an acid sensation

White wine vinegar directly into 50 lL bottle,

adding 10 mL H2Oa
Dissolve 60 lL acetic acid to 10 mL with H2Oa

Rancid

Butyric (in excess)

Rancid

Smelly sensation reminiscent of butyric acid

Smelly sensation, characteristic of fatty substances,
following deterioration reactions (rancidness)

To 15 lL butyric acid add 10 mL vaseline oila

5 g D.O.P Idiazabal cheese (>12 months ripening).
Slice and keep at room temperature for about
6 days

Animal

Cowshed

Smelly sensation reminiscent of an unventilated

sheep cowshed
Smelly sensation reminiscent of substances presenting
faeces and fermentation products with enteric bacteria
Smelly sensation reminiscent of deteriorating protein
rich foods caused by microbial growth, leading to
unpleasant odours

5 g rear sheep wool

Elementary taste perceived in the mouth with watery

solutions of diverse organic substances such as
lactic acid
Flavour sensation reminiscent of white wine vinegar

Add 65 g curd to 35 g natural yoghurt

Odour
Acid

Faecal
Putrid

Flavour/aftertaste
Acidic

Acid (in excess)

Vinegar
Animal

Cowshed
Natural rennet
(in excess)c
Faecalc
Putridc

Bitter
Pungent

Abrasive
Pungent

Salty (in excess)

a
b
c

5 g sheep dung
5 g putrid cheese. Add 3.5 mL BS to 100 g curd

To 100 g curd add 3.3 mL white wine vinegar

Flavour sensation reminiscent of a sheep cowshed

Flavour sensation associated with lamb rennet paste
in excess
Flavour sensation reminiscent of substances present
in faeces and fermenting products in enteric bacteria
Flavour sensation associated with deteriorating protein
rich foods, caused by microbial growth, leading to
unpleasant odours

5 g rear sheep wool

To 0.5 g rennet paste add 35 mL H2O.
Stir for 15 min and lterb
5 g sheep dung

Elementary taste produced by diluted watery solutions

of diverse substances such as quinine or caeine

To 0.35 g anhydrous caeine add 50 mL H2O.

Add 4 mL BS to 100 g curd

Irritating action perceived in the mouth and pharynx

mucous, possible accompanied by a sensation of warmth
Sensation perceived in the mouth as an irritation,
burning or stinging

To 5 mL distilled alcohol add 15 mL H2O.

20 mL BS
Add 50 mL boiling H2O to 0.5 g cayenne
pepper seeds. Keep for 5 min and lter.
2 mL oating add to 100 g curd

Elementary taste produced by diluted watery solutions

of diverse substances such as sodium chloride

To 2.35 g anhydrous sodium chloride

add 50 mL H2O. Add 3.5 mL BS to 100 g curd

5 g putrid cheese

10 mL of base solution (BS).

15 mL of base solution (BS).
References for avour/aftertaste, natural rennet, faecal and putrid, are not ingested. Prepared and presented as odour references.

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

432

representatives of each of the parameters. Once the panelists acquire sucient theoretical knowledge, they participate in two types of training sessions:

are conducted: qualitative description of characteristics

and defects; and quantication on scale of overall parameters (odour, texture, avour and aftertaste).

1. Description of appearance: Panelists describe cheeses in

qualitative terms, taking into account both the optimum
characteristics and possible defects.
2. Evaluation of appearance: quantication on scale (shape,
rind, paste colour, and eyes).

3.3.3. Training by harmonization tasting

Prior to each harmonization session, panelists undergo a
refamiliarization (i.e. reminder) session to refresh their
knowledge of the dierent characteristics and/or possible
defects of PDO Idiazabal cheese using the references.
Afterwards, they evaluate the dierent product samples.
In these harmonization sessions, panelists discuss openly
among themselves, with the intervention of the panel leader
as moderator.

3.3.2. Training for odour, avour, aftertaste and texture

The panel leader gives a theoretical description of each
of the olfactory-taste and texture characteristics and
defects. The use of standardized references is an essential
point in order to ensure that the panelists memorize these
sensations (Tables 58).
Once the panelists have acquired the necessary knowledge on odour, avour, aftertaste and texture descriptors
and defects, dierent samples of PDO Idiazabal cheese
are presented and two types of training session procedures
Table 8
Denition and references for defects in texture
Defect

Denition

Reference

Elastic

Sample quickly recovers its initial thickness

after being compressed and deformed by the
thumb
Easily deformed when compressed between
the tongue and palate
Resilient to being cut or compressed when
teeth clenched
After masticating, rounded, very ne sized
particles noted in the mouth
After masticating, separated, large sized,
ne sized particles noted in the mouth
After masticating, separated, large sized
particles are noted in the mouth that is
more or less hard on the teeth
After masticating, angular shaped separated
particles noted in the mouth which, when
crushed, let out an audible crunch
Lack of smoothness and unpleasant
sensation perceived as the sample comes
into contact with the surface of the
mouths soft tissues
Forms a paste with saliva and continuously
melts giving a perception of presence
in the mouth
Releases some amount of liquid following
mastication
Releases a fair amount of liquid following
mastication
Cohesive when mixed with saliva, adheres
to the surface of the soft parts of the mouth
Slowly deforms in the mouth before
breaking up, without recovering its
initial shape
Easily breaks into pieces when the sample
is compressed with ngers or incisors
Absorbs saliva
Dissolves in contact with saliva and
disappears in the mouth

Frankfurter
sausage

Soft
Hard
Floury
Sandy
Lumpy

Crystals

Coarse

Melting

Moist
Liquid
Doughy
Plastic

Brittle
Dry
Soluble

Melted cheese
Raw carrot
Potato puree
Breadcrumbs
Medium
grained
couscous
Sugar

Pippin apple

Melted cheese

Boiled white
of egg
Granny Smith
apple
Boiled egg yolk

3.4. Preliminary individual qualication of panelists:

PDO Idiazabal cheese as an example
As a preliminary stage of qualication for quality quantication on scale, a study of the discriminating capacity,
repeatability and reproducibility for each panelist is carried
out in two sessions. The sensory analysis laboratorys
(SAL) criteria are the following:
Discriminating capacity: a panelist is qualied in terms
of discriminating capacity if a compatibility rate greater
than 1 is overcome (as a value of uncertainties, the reference value of standard deviations is taken) in 50% of
the discriminating parameters of the values in the two
sessions as a whole. The compatibility rate is dened as
jx1  x2 j
I c q
I 21 I 22
taking as I1 and I2 the reference values.
Repeatability: the standard repeatability deviation is calculated as the average of the standard deviations in the
session with each cheese. A person is considered repeatable when the standard repeatability deviations are less
than or equal to the reference value in 50% of the total
number of parameters analysed.
Reproducibility: the standard reproducibility deviation is
calculated as the square root of the sum of the variances
between sessions plus the variance due to repeatability. The variance between sessions is considered as the
variance corresponding to the average of the values
recorded in the sessions. A panelist is considered as
qualied in reproducibility when in 50% out of the total
number of parameters, the standard reproducibility
deviation value is less than or equal to the reference
value taken.

Soft toee

Toast
Toast
Sugar

As an example, Table 9 shows the qualication results of

one of the judges. During two sessions (eight cheeses per
session), panelists evaluated four dierent Idiazabal cheeses
(ICh), where two of them were evaluated three times in each
session. In this example, results show that this panelist is
qualied to reproducibility, repeatability and discriminat-

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

433

Table 9
Means, standard deviation, reproducibility, repeatability and discriminating capacity of one panelist
Shape

Rind

Paste colour

Eyes

Texture

Odour

Flavour

Aftertaste

Panelist: reproducibility and repeatability

Day 1
ICh 1 ICh 2 ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

Mean (N = 3)
S sample

3.25
0.50

3
0

3.75
0.50

2
0

4
0

3
0

3
0

1
0

4
0

4
0

4
0

4
0

4
0

3.50
0.58

4
0

3.25
0.50

Day 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

Mean (N = 3)
S sample

3
0

4
0

4
0

2
0

3
0

3
0

3
0

2
0

5
0

4
0

4.25
0.5

3
0

4
0

2.25
0.50

4
0

2
0

S
S
S
S
S

0.50
0
0.25
0.35
0.43

0
0
0
0
0

0.50
0
0.25
0.35
0.43

0
0
0
0
0

0
0
0
0
0

0
0
0
0
0

0
0
0
0
0

0
0
0
0
0

0
0
0
0
0

0
0
0
0
0

0
0.50
0.25
0.35
0.43

0
0
0
0
0

0
0
0
0
0

0.58
0.50
0.54
0.05
0.54

0
0
0
0
0

0.50
0
0.25
0.35
0.43

session 1
session 2
repeatability
between session
reproducibility

% Reproducibility
% Repeatability

94
94

Panelist: discriminating capacity

ICh 1 ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

Mean
Ic

3.12
0.53

3.87
2.65

3.5
0.70

3
2.12

1.5

4.5
0.70

4.12
0.88

3.5

4
1.59

2.87

4
1.94

2.62

% Discriminating

67

3.5

Panel as a whole: discriminating parameters

ICh 1 ICh 2 ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

ICh 1

ICh 2

Mean (N = 10)
S panelists

3.10
0.22

1.61
0.49

3.37
0.46

2.43
0.72

3.07
0.33

1.29
0.33

3.77
0.57

3.45
0.38

4.27
0.42

3.82
0.43

3.77
0.58

2.96
0.46

3.86
0.76

2.97
0.40

t calculated
t tabulated

1.09
2.23

3.36
0.73

3.56
0.72
7.06
2.10

3.47
2.10

11.97
2.10

OK

OK

OK

1.49
2.10

2.35
2.10

3.45
2.10

3.25
2.14

OK

OK

OK

ICh 1: Idiazabal cheese 1; ICh 2: Idiazabal cheese 2; N: Number of times that every kind of cheese is evaluated during each session; S: Standard deviation;
Ic: Compatibility race.

ing capacity. That is, 94% of standard reproducibility and

repeatability deviations are less than the 0.5 reference value;
and the compatibility rate is greater than 1, form more than
2/3 of the discriminating parameters. In the example (Table
9), shape and texture parameters were not considered discriminating parameters by the panel as a whole.
3.5. Qualication of panelists as experts: PDO Idiazabal
cheese as an example
Qualication of panelists as experts (quality quantication on scale and defects identication) is done on the basis
of their performance history. The competence of panelists
in formation is measured after a minimum of 10 sessions
with eight cheeses each (80 cheeses).
3.5.1. Qualication for quality quantication on scale
A comparison is carried out between the panelist rating
and the average rating obtained from the qualied panel as
a whole. The panelist is qualied for quality quantication
on scale if during 10 consecutive sessions the panelist shows
less than 15% of deviations (dierence with the average of

the qualied panel bigger than 1) considering all the

parameters evaluated in all the cheeses (eight cheeses per
session eight parameters per cheese = 64 comparisons
per session).
3.5.2. Qualication for defect identication
A comparison is carried out comparing defects identied
by each panelist with those defects identied by the qualied panel. The SAL has established that a cheese presents
a defect when at least ve out of the seven qualied panelists point this out. It is considered that a panelist is qualied to identify a specic defect if he/she shows a
competence according to the following criteria:
Number of cheeses with the defect that the panelist has
evaluated and has pointed out/total number of cheeses
with said defect evaluated by the panelist 100 >
66.6%.
For example, when a defect has appeared on three occasions, then the panellist is considered to be qualied when
he/she has noted the defect at least twice out of the possible
three times. When a panelist is qualied for more than

434

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

66.6% of the total number of valuable defects, he is

acknowledged to have the capacity to join the PDO Idiazabal cheese panel as an expert. In order to reinforce their
performance, harmonization tasting procedures are undertaken, at two-week intervals to increase the historical experience in identifying defects of the previously qualied
panelists.
Experience has shown us that at least one year of activity is needed for a panelist to demonstrate sucient historical experience to be considered qualied as an expert for
evaluating the quality of this complex specic product.
This long period is necessary due to the limitations in food
sensory analysis, such as the lack of stable references and
sensory fatigue.
3.6. Sensory quality evaluation method: PDO Idiazabal
cheese as an example
Sensory evaluation is carried out in standardized
booths. Sensory sessions usually starting at 10:00 h, and
may extend for a maximum of 2 h. In the case of the sensory quality evaluation of Idiazabal cheese, the maximum
number of test samples to be evaluated in each session is
8, being the temperature of the samples 16 2 C. Samples
are prepared according to the specic procedure published
by Lavanchy et al. (1993).
Sample replication is not considered in routine tests.
When a well-trained panel generates data with considerably lower variability and replications are also desirable,
but not essential, and if replications cannot be accommodated, the data is nonetheless still valid (Moskowitz
et al., 2003).
3.6.1. Tasting room preparation
Prior to each session, checks are made on the temperature (21 2 C) and relative humidity (60 20%) in the
booths, and on the correct working order of the taps and
warning devices. In each booth, a paper serviette, a plastic
cup, a bottle of water with low mineralization and a solid
food (apple, biscuit without salt) are placed.
3.6.2. Conditions for panelists
Samples are to be analysed by seven of the expert qualied judges for this specic product. In general, a small
number of panelists are adequate if they are well-trained
(Moskowitz et al., 2003).
Prior to sensory evaluation panelists should take the following precautions into account: personal hygiene and
clothing, punctuality; to arrive at least 30 min before the
test. Panelists should abstain from consuming any strongly
avoured products and from using cosmetics with very
intense or residual odours, and should also be rested and
clear-headed. Panelists should not take part in the test if:
pregnant, suering from a headache, nasal congestion or
using drugs. If judges nd themselves in any of these
conditions, they should advise the panel leader 24 h in
advance.

3.6.3. Scales of evaluation

Quality evaluation is conducted by taking into account
the extent to which each sample complies with the optimum state of PDO Idiazabal cheese in each of the eight
parameters analysed (shape, rind, paste colour, eyes,
odour, texture, avour and aftertaste). A 7-point category
scale is used, where 1 = null, meaning that the sample is
considerably removed from the optimum state for each
parameter; 7 = maximum, full compliance with the optimum state. To carry out evaluation, each judge has documents in the test booth showing: the optimum states for the
dierent parameters (Table 1); the denition, evaluation
technique and references used in characteristics (Tables 5
and 6) and defects (Tables 7 and 8) of olfactory-taste and
texture; a catalogue of photographs illustrating the optimum characteristics (Table 1) and possible defects (Table
4) of appearance; and the evaluation criterion (Fig. 2).
3.6.4. Procedure
Prior to sensory evaluation, a harmonization tasting
with one or two samples of PDO Idiazabal cheese is carried
out. These samples are to be dierent from the routine samples. In order to avoid problems with order and presentation, this order shall be randomized and dierent for
each judge. In the test booths, communication between
the judges and the panel leader is done by a system of
warning lights.
Panelists giving points below the average value (<4 on
the scale) mark the identied listed defect/s. Evaluating
allows noting down any defect that, despite not being
listed, has been identied by any of the judges for any of
the samples. These new defects will not be taken into consideration when drawing up the test reports for the current
year. In order to include a new defect on the list, the number of judges coinciding on the same defect, in the same
cheese and at the same session, should be equal to or
greater than 5. Such defects on the list that have not been
recorded in any test report in the course of a year are
removed from the following years list. Depending on the
extent to which the sensory parameters appearing in Table
1 are complied with, and the possible presence of defects,
panelists are to give points to the samples in line with the
criteria shown in Fig. 2.
Test reports are according to ISO 17025 and they are
sent to the protected designation of origin (PDO) Idiazabal
cheese, including the average points given by the panel for
each parameter and any defect detected. Nowadays, these
results permit the Regulatory Council of Idiazabal cheese
to certify, from the sensory viewpoint, the identity and
quality of the products made by 104 registered cheese producers. To do so, this Certication Body establishes the
minimum limits for quality in terms of points and the seriousness of the possible defect (not all the parameters are
considered equally important). It is important the panelists
do not know where the limits laid down by the Designation
stand, so that they will not be inuenced by factors of a
psychological nature. This system enables producers to

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

435

Tree diagram for ascribing points

To be applied for each parameter and each cheese

Do you note-detect any defect to be pointed out?

No

Yes

To consider all the characteristics

Do you consider this/these defect/s to be slight?

To be all present and appropriate?

No

Yes

No

Yes

7
One absent-inappropriate?

No

3
Do you consider this/these defect/s to be medium?

Yes

No

Yes

6
Various absent-inappropriate?

No

2
Do you consider this/these defect/s to be serious?

Yes

Yes

Are the majority absent-inappropriate?

Yes

4
Fig. 2. Tree diagram for evaluating sensory quality (evaluation criteria).

detect points to be improved upon in the minimum sensory

quality when making their product.
Munoz, Civille, and Carr (1992) discuss dierent
approaches to sensory quality assessment. A reasonable
compromise between the quality-rating method and a
descriptive approach was proposed by Beckley and Kroll
(1996). The centrepiece of this procedure is a scale for overall quality. The advantage of this method is its outward
simplicity in using an overall rating and recognizing that
there are situations where products will not exactly match
the gold standard but are still acceptable to ship.
In the future, in spite of the open discussion between
those in favour or otherwise of sensory quality assessment
by experts, accreditation will become the tool for evaluat-

ing the validity of the methods applied in sensory

laboratories.
3.7. Validation of the method: PDO Idiazabal cheese
as an example
In a similar manner to the preliminary individual qualication of panelists for quality quantication on scale, the
repeatability, reproducibility and discrimination capacity is
considered as the validation criteria of the method.
Maximum uncertainties of repeatability and reproducibility acceptable by the SAL should be previously established. These values can be calculated from three sessions
with three dierent Idiazabal cheeses, where every panelist

Mean (N = 14)
S panelists
t calculated
t tabulated

Idiazabal
cheese

Other PDO cheese

4.14
0.86
2.95
2.064

3.36
0.50

OK
ICh 1: Idiazabal cheese 1; ICh 2: Idiazabal cheese 2; S: Standard deviation;
N: Number of times the sample has been evaluated.

0.20
0
0.10
0.43
0.44
3,43
2.86
0.10 3.71
0.10 2.86
0.10
0.42
0.43
3.71
3
0.10 3.86
0
3.14
0.05
0.37
0.38
3.57
2.86
0
3.43
0
2.86
0
0.33
0.33
3.57
3
0
3.57
0
3
0
0.25
0.25
2
2.43
0
2
0
2.43
0
0.08
0.08
2.86
2.71
0.40 2.86
0.20 2.71
0.30
0.27
0.41
4.57
4

Aftertaste
Flavour

ICh 1 ICh 2 Ss
ICh 1 ICh 2 Ss

Odour
Texture

ICh 1 ICh 2 Ss
ICh 1 ICh 2 Ss

Eyes
Paste colour

ICh 1: Idiazabal cheese 1; ICh 2: Idiazabal cheese 2; Ss: standard deviation of each session; Sbs: standard deviation between sessions.

Discriminating capacity

0
4
0.20 4.28
0.10
0.15
0.17

0.258

3.14
3.43

S session
0.142
0.082
0.082
0.103
0.237

3.14
3.14

ICh 3

Mean day 1
Mean day 2
S repeatability
Sbs
S reproducibility

ICh 2

3.43
3.57
4
4
4.14
4
S repeatability
S between
session
S
reproducibility

ICh 1 ICh 2 Ss

ICh 1
3.71
3.86
4

Rind

Session 1
Session 2
Session 3

ICh 1 ICh 2 Ss

Repeatability and reproducibility

Table 11
Quality control for quantication on scale of sensory parameters

Table 10
Example of results of validation (mean of the seven panelists as a whole)
for the parameter avour

ICh 1 ICh 2 Ss

evaluates each Idiazabal cheese sample three times. After

this study, the SAL have proposed 0.5 (repeatability) and
0.8 (reproducibility) as maximum uncertainties accepted.
From these criteria, validation of the method (reproducibility and repeatability) is carried out in three later sessions,
in the same way as the uncertainties study was done. The
discrimination capacity is analysed in a later session where
each panelist evaluates both a sample of Idiazabal cheese
and a sample of non-Idiazabal cheese, doing each twice.
As an example, Table 10 shows the validation results for
the parameter avour. This parameter can be validated
because the repeatability and reproducibility values
obtained by the qualied panel are less than the maximum
uncertainties acceptable proposed by the SAL. Also, the
student-t test shows the discriminating capacity of the
method for the parameter avour.
In a similar manner a historical analysis and validation
(repetition of cheeses with defects) has been carried out for
the identication of defects by the panel as a whole.
Prociency/accreditation is a very current and pressing
issue being addressed by sensory professionals worldwide.
The SAL validation dossier also includes prociency tests
with other laboratories. Prociency testing is the use of
inter-laboratory test comparisons as part of the process
to assess the ability of laboratories to competently perform
test for which accreditation is held. Nowadays, prociency
testing schemes in Sensory Science are available (Moskowitz et al., 2003). The rst attempt to address sensory prociency was initiated in 1999 by the PROFISENS
European Union project, where technical requirements
for conducting sensory prociency tests and the management systems requirements are provided (Lyon, 2001).
Other relevant publications of this project are the papers

ICh 1 ICh 2 Ss

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

Shape

436

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

by McEwan, Hunter, Van Germert, and Lea (2002b,

2002a) that describe measuring performance panel issues
for ranking and prole testing.
The SAL has established systematic quality controls for
maintaining both method validity and the individual qualication of the panelists.
3.8. Quality control activities: PDO Idiazabal cheese
as an example
3.8.1. Maintaining method validity
Quality control is conducted by studying the reproducibility and repeatability of the panel as a whole, in order
to maintain method validity. To do so, at two monthly
intervals, a cheese is analysed, in duplicate, at the same session and afterwards at a subsequent session. The standard
deviations for repeatability and reproducibility for each
parameter are calculated. These deviations should be less
than the SALs maximum acceptable uncertainty for more
than 80% of the parameter analysed. As example, Table 11
shows quality control activity for the quantication on
scale carried out by the SAL in 2005. This control was positive because both standard deviation for repeatability and
reproducibility were less than SALs maximum acceptable
uncertainties in 100% of the parameters analysed.
On an every two-week basis, quality control for the
identication of defects is carried out using a systematic
repetition of cheese samples, where defects had been iden-

437

tied in previous routine sessions. If the results of repetition show fewer than ve members of the panel to
identify the defect, then it will be openly discussed to
achieve consensus as to whether or not the specic defect
is found in the cheese. To back this up, by means of harmonization tasting, cases where panelists were not unanimous
(34 panelists noting down the defect) are investigated in
order to reduce the frequency of these situations as much
as possible.
3.8.2. Follow-up and monitoring the panelists
To ensure the individual qualication of the panelists
and the panel as a whole, it is essential to follow up and
monitor each one of the expert judges and the panel. To
do so, the SAL has standardized procedures:
Follow-up and routine monitoring of the panelists. An
individual report is drawn up comparing the results
given by each the judges in each session with the average
results from the panel as a whole. In those cases where
the standard deviation of the panel is P1, an analysis
of the data from panelists responsible of this dispersion
is conducted. Panelists who stand at a distance from the
rounded o average obtained by the panel as a whole by
one unit are noted as a deviation for these parameters.
This report is drawn up for each session, and is to be
shown to the judges so that they can see where there
were deviations. Where a panelist, in a given session,

Table 12
Individual report of a panelist after a session
Data

Judge

Cheese code

Shape

Rind

Paste colour

Eyes

Texture

Odour

Flavour

Aftertaste

256
Deviation?
Panel mean

5
Yes
3

4
Yes
2

789
Deviation?
Panel mean
123
Deviation?
Panel mean
951
Deviation?
Panel mean

632
Deviation?
Panel mean

6
Yes
4

701
Deviation?
Panel mean

No. cheeses:
No. parameters:
No. deviations:
% Deviations:
No. deviations:
% Deviations:

6
48
3
6.25
4
8.3

Panelist
Panel

Defects
Defects identied by the panel:
Defects not identied by the panelist:
Defects identied by the panelists:
% Identied:
82.14

28
5
23
OK

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

438

presents more than 15% deviations for all the parameters evaluated, one of the following measures is taken:
the panelist is interviewed to evaluate any possible
causes, re-training on the characteristics aected, temporary withdrawal and/or personal re-qualication. As an
example, Table 12 shows the individual (each panelist)
report appearance after a session where six cheeses were
analysed.
Follow-up and routine monitoring of the panel. If during a
given session the panel as a whole presents 15% deviations for all the attributes evaluated, then the following
measures are to be taken: re-training of the panel and/or
temporary withdrawal from participating.

panelists), then he/she should be subject to: re-training

and/or temporary withdrawal.

In order to ensure maintaining the qualication of

defects, a follow-up and monitoring of the rate of defects
identied by each panelist at each session is carried out.
If for the total of cheeses in a session, the rate of defects
noted by a panelist is less than 66.6% of the total number
of existing defects (noted by more than ve out of the seven

All technical activities should be developed in line with a

quality management system. The management requirements under the ISO 17025 standard should be reected
in a manual and/or in quality procedures (Table 13). This
system of documenting and recording makes it possible
to demonstrate to the accreditation body that the standards are complied with in the external audits (or third
party audits). The system for dealing with internal nonconformities and claims by correctional and/or preventive
actions, internal and external audits and by the management reviewing the system, are the most dynamic activities
of the quality system management and they make it possible to improve the activities at SAL.

Table 13
Cross-reference table showing ISO 17025 Standard items and the quality
manual (QM) and operative quality procedures (OQP)
ISO 17025:1999
standard section

QM

1. Object and eld of

application
2. Standards for consulting
3. Terms and denitions

01

4. Management requirements
4.1. Organization
4.2. Quality system
management
4.3. Monitoring documents

4.4. Reviewing applications

oers and contracts
4.5. Subcontracting tests
and calibrations
4.6. Out pooling services
and supplies
4.7. Customer service
4.8. Claims
4.9. Non-conformities
control
4.10. Corrective actions
4.11. Preventive actions
4.12. Records control
4.13. Internal audits

OQP

3.9. Management system and improving activities

4. Conclusions

02
03
04

OQP-01: Drafting OQP

OQP-02: Drafting NWP*
OQP-03: Monitoring
documents
OQP-08: Acquisition and
inventory of supplies and
evaluation of suppliers

OQP-12: Customer service

and claims management
OQP-10: Treatment of nonconformities, correctional and
preventive actions

OQP-06: Control of internal

records and test reports
OQP-11: Conducting internal
audits and management reviewing
the system

4.14. Reviews by the

management
*

3.8.3. Panel re-training: specicity of the PDO Idiazabal

cheese
Idiazabal cheese is a seasonal product, made from January to July. Sensory evaluations are also seasonal (FebruaryOctober). During the period from November to
January, panelists are called to carry out some of the specic training activities mentioned above.

NWP: normalized work procedure (technical procedure).

Ensuring the food quality on a sensory basis is an

important corporate goal in a competitive environment.
Sensory evaluation is a powerful resource with a great variety of applications in the commercial sector. The standardization and accreditation of sensory evaluation methods
for the quality certication of food products is a pressing
need, particularly with regard to those with protected designation of origin (PDO). This study proposes a general
approach for the development of an accredited sensory
method for the quality evaluation of food and beverages,
taking into account specic characteristics and possible
defects in the products. The work shows the application
with a specic product (Idiazabal cheese) for example.
Also, it shows the selection, training and qualication
required for expert panelists for implementation. An
important activity is the systematic control quality to
assure the maintenance of expert panelist qualication
(individually) and of the previously validated method. In
terms of PDO products, this manuscript would make possible to respond to an existing need: to show a work methodology so that the members of expert panels (Sensory
Committees) can carry out an objective and independent
task. The proposed method allows the application of good
sensory practices and ensures consistent data.
The approach proposed may be transferred to any type
of food as a reference for the accreditation of sensory quality evaluation methods according to ISO 17025, especially

F.J. Perez Elortondo et al. / Food Quality and Preference 18 (2007) 425439

for sensory certication activities. Each product should

dene its quality standard and, on the basis of this standard, the sensory specialist should carry out the sensory
evaluation using a panel specically trained for this purpose. The approach is well suited to standard commodities
where minimum levels of quality could be ensured, but
excellence is rarely the issue. This proposal does not
exclude applying other sensory analysis methods for other
purposes (hedonic and descriptive tests) to complement it.
Quality evaluation methods should take the data derived
from the application of_hedonic and descriptive tests.
Acknowledgements
Financial support of the work was provided by Consejo
Regulador de la Denominacion de Origen Queso Idiazabal,
the Departamento de Agricultura y Pesca del Gobierno
Vasco and the Universidad del Pas Vasco/Euskal Herriko
Unibertsitatea (project 9/UPV 00042.125-15317/2003 and
Quality Chair nancement). Expert Members of the
Tasting Committee of the Regulating Council of the Designation of Origin are thanked for their enthusiastic participation in this study.
References
Beckley, J. P., & Kroll, D. R. (1996). Searching for sensory research
excellence. Food Technology, 50, 6163.
Bertozzi, L., & Panari, G. (1993). Cheeses with appellation dOrigine
Controlee (AOC): Factors that aect quality. International Dairy
Journal, 3, 297312.
DOCE (1992). Reglamento 92/2081/CEE del Consejo relativo a la
proteccion de las indicaciones geogracas y de las denominaciones
de origen de los productos agrcolas y alimenticios, Diario Ocial de
las Comunidades Europeas de 14 de julio de 1992.
ENAC (2003). Gua para la acreditacion de laboratorios de analisis
sensorial. G-ENAC-02, Rev. 1. Octubre 2003. Entidad Nacional de
Acreditacion, Madrid, Espana.
I.S.O. 5495 (1983). Sensory analysis. Methodology. Paired comparison
test. International Organization for Standardization, Gene`ve,
Switzerland.

439

I.S.O. 4120 (1983). Sensory analysis. Methodology. Triangular test.

International Organization for Standardization, Gene`ve, Switzerland.
I.S.O. 6564 (1985). Sensory analysis. Methodology. Flavour prole
methods. International Organization for Standardization, Gene`ve,
Switzerland.
I.S.O. 6658 (1985). Sensory analysis. Methodology. General guidance.
International Organization for Standardization, Gene`ve, Switzerland.
I.S.O. 10399 (1991). Sensory analysis. Methodology. Duo-trio test.
International Organization for Standardization, Gene`ve, Switzerland.
I.S.O. 8586 (1993). Sensory analysis. General guidance for the selection,
training and monitoring of assessors. Part 1: Selected assessors.
International Organization for Standardization, Gene`ve, Switzerland.
I.S.O. 1036 (1994). Sensory analysis. Methodology. Texture prole.
International Organization for Standardization, Gene`ve, Switzerland.
I.S.O. 17025 (1999). General requirements for the competence of testing
and calibration. International Organization for Standardization,
Gene`ve, Switzerland.
Issanchou, S., Schlich, P., & Lesschaeve, I. (1997). Sensory analysis:
methodological aspects relevant to the study of cheese. Lait, 77, 512.
Lavanchy, P., Berodier, F., Zannoni, M., Noel, Y., Adamo, C., Squella,
J., & Herrero, L. (1993). Levaluation sensorielle de la texture des
fromages a` pate dure ou semi-dure. Etude interlaboratoires. Lebensmittel Wissenschaft und Technology, 26, 5968.
Lawless, H. T., & Heymann, H. (1998). Sensory evaluation of food:
principles and practices. New York: Chapman and Hall.
Lyon, D.H. 2001. International guidelines for prociency testing in
sensory analysis. Guideline No. 35. CCFRA (Campden & Chorleywood Food Research Association), Chipping Campden, GL55 6LD,
United Kingdom.
McEwan, J. A., Heinio, R., Hunter, E. A., & Lea, P. (2002a). Prociency
testing for sensory ranking panels: measuring panel performance. Food
Quality and Preference, 14, 247256.
McEwan, J. A., Hunter, E. A., Van Germert, L. J., & Lea, P. (2002b).
Prociency testing for sensory prole panels: measuring panel performance. Food Quality and Preference, 13, 181190.
Moskowitz, H. R., Munoz, A. M., & Gacula, M. C. (2003). In viewpoints
and controversies in sensory science and consumer product testing.
Trumbull, Connecticut: Food and Nutrition Press, Inc.
Munoz, A. M., Civille, G. V., & Carr, B. T. (1992). Sensory evaluation in
quality control. New York: Van Nostrand Reinhold.
Perez Elortondo, F. J. (1993). Actividades tecnicas y de control de la
Denominacion de Origen Idiazabal. Sustrai, 28, 4651.
Perez Elortondo, F. J., Barcenas, P., Casas, C., Salmeron, J., & Albisu,
M. (1999). Standardization of sensory methodologies: some applications to protected designation of origin cheeses. Sciences des Aliments,
19, 543558.