Services Provided by LCBO

Quality Assurance

Leading Sensory Evaluation

Services

LC Quality Assurance

VQA Appellation System

VQA Mandate
Appellation of Origin System
Quality Standard
Control of use of specified terms,
descriptions and designations
associated with the VQA appellation
system

LC Quality Assurance

Role of Sensory Evaluation

Testing
Quality of Beverage Alcohol
Products

Sensory Quality
Chemical Composition
Microbiological Stability
Packaging and Labelling
Standards

LC Quality Assurance

Role of VQA Sensory

Evaluation Testing
Scope

Free of technical faults / defects

Typicity of varietal character for varietal wines

A wine bearing varietal designation shall be

assessed to determine if the varietal designation
for the wine exhibits the predominant character of
a wine produced from the designated grape
variety/varieties (VQA Rules)

Typicity of the wine category: Late Harvest,

Icewine, Nouveau, Sparkling Traditional Method,
Icewine Dosage, Botrytized Wine (VQA Rules)

LC Quality Assurance

Role of VQA Sensory

Evaluation Testing

A wine shall be deemed to have passed

the taste test if a majority of the members
of the Tasting Panel determine:
That the wines attributes fairly reflect the
viticultural and oenological quality standards
established in O.Reg. 406/00 (Rules) without
defects or flaws; and
That the wine is representative of quality
wines of the stated category (VQA Rules)

LC Quality Assurance

Sensory Panels

Grading Panel 4 panel

groups of 5 panellists
each

28 Members (20 regular

panellists, 8 alternate
tasters)

LC Quality Assurance

Panel Member Selection

Criteria
Panel members are LCBO
Product Consultants who
work in retail stores within
metropolitan Toronto area
Continuous training/
development
opportunities
LCBO Product Knowledge
I, I, III
Certification from the
Wine and Spirit Education
Trust
Master of Wine
Quality Assurance
Certification

LC

Panel Member Selection

Criteria
Panel members are subjected
to annual :
Sensory evaluation testing
Product knowledge testing
25 % of the questions are
VQA specific

Demonstrated professionalism
Sensory Evaluations Code of
Conduct

LC Quality Assurance

Panel Groups Assembly

Criteria:

Consistent
Performance
Reproducible
Results
Balanced
Panels

Results from the annual

testing
Testing performance history
Previous experience (as a
Grading Panel member, from the
industry)

LC

Performance monitoring
data
Quality
Assurance

Panel Training
Training Elements
Aroma, flavour recognition
Product category recognition
Varietal character recognition
Regional character recognition
Winemaking techniques
Technological influences
Defect identification

LC Quality Assurance

Sample Presentation

Blind sample presentation:

Samples are presented

without identifying
markings
Uniform sample
Sample information: Varietal composition,
presentation
vintage year, wine category, method of
production (where applicable: sparkling wines,
icewine dosage, etc)

LC Quality Assurance

Sensory Evaluation Method

Sensory Evaluation Grading
System

LC Quality Assurance

Sensory Evaluation Grading

System

Applicabilit
y

Ob
jec
tiv
ity

Client
Satisfaction

fi
Ef

cy
n
e
ci

Superior objectivity system

design eliminates bias
Wider system applicability
system design equally applies
to all wine categories
Efficient and effective
calculation of results time
savings, accuracy
Data Analysis Tools &
Traceability of results
enabled by automated data
collection and quantification of
the results

LC Quality Assurance

Sensory Evaluation Grading

System
Sensory evaluation
grading system elements:

New sensory evaluation

grading method
Integrating technology:
Automated data
collection
Automated data
quantification, analysis
and reporting

LC Quality Assurance

Sensory Evaluation Grading

System
Sensory characteristics applicable to
all types of beverage alcohol products.

Appearance and Colour

Aroma (primary and secondary) and Bouquet
Taste
Harmony
Appearance

Harmony
Colour

Aroma

LC Quality Assurance

Bouquet
Taste

Sensory Evaluation Grading

System
Sensory characteristics and attributes are
evaluated using five grading categories.
Categor
y

Excellent

Very
Good

Quality Outstandin Superior/

g
Very
Level
/Exception
Correct
al

Good

Satisfactor
y

Unsatisfacto
ry

Typical

Weak/Not
at full
potential

Faulty /
Defective

LC Quality Assurance

Grading Form
Characteristic
s & Attributes

Grading Categories
Excelle
nt

Very
Good

Appearance &
Colour

Aroma
&
Bouque
t

Correctn
ess

Intensit
y

Taste

Good

Quality

Correctn
ess

Intensit
y
Finish

Satisfact
ory

LC Quality Assurance
x

Unsatisfac
tory

Comme
nts

Innovative Technology

Data
Data
Data
Data

collection
quantification
analysis
reporting

LC Quality Assurance

Data Collection Tablet PCs

Automated
Interactive
Flexible-Wireless
Pen or/and keyboard
Handwritten notes

LC Quality Assurance

New system implemented on June 1 2004; Comparison for periods June December 2003 and 2004
45
40

Scores
Frequency, %

Scores Frequency, 2003

Scores Frequency, 2004
Difference 04-03
2004

35
30

2003

25
20
15
10

5.9

5.7

4.5

3.5
0.3

1.6

0
-5

<12.0

12.0 - 12.4
-3.2

12.5 - 12.9
-3.0

13.0 - 13.4

13.5 - 13.9

14.0 - 14.4

14.5 - 14.9
-4.0

-10
-15

-11.4

LC Quality Assurance

15.0 - 15.4

15.5 - 15.9

>16.0

Scoring Intervals
(0-20 Scoring Scale)

QA of Sensory Evaluation
Results
Quality Sensory Evaluation Result

Quality
Assurance
System
Controls

Verification
Tasting

Performance
Monitoring

LC Quality Assurance

Proficiency
Programs

QA of Sensory Evaluation
Results
VQA Tasting protocol (VQA Rules)
2nd Bottle tasting

Grading System Controls:

Security of panellist registration

Forced completeness of the assessment
Ratings cannot be changed, once finalized
Sample evaluation cannot be redone, once
completed
Validation questions in the grading questionnaire
Calculation validation based on statistical
measures
Quality Assurance
Automated calculation
and reporting

LC

QA of Sensory Evaluation
Results

Verification Tasting - an independent sensory

evaluation of the products in conditions identical to
the tasting panel.
The verification process has a quality assurance role
and provides a reference value that is then
compared with the panel results to identify any
discrepancies in the assessment.
Verifiers qualified Quality Assurance tasters. A
tasting session may require up 2 Verifiers.
Verifiers results are typically not included in the
calculation, unless discrepancy situation.

LC Quality Assurance

QA of Sensory Evaluation
Results
Panellists performance is monitored for each
tasting session
Measures:
Outliers Frequency, % - measure of rating
accuracy
Rating Rank measure of rating bias (high,
low, trends)
Sensitivity data - missed defects, good
detection, hyper-sensitivity
Feedback on performance Feedback Report

LC Quality Assurance

QA of Sensory Evaluation
Results
Panellist Feedback
Report

Tasting Session:
Panellist:
Total Number of Samples:
Total Number of Outliers:
Outliers Frequency, %
Rank, Descending

4-Apr-2005 2:23 PM
Panellist Name

26
4

15.4
2.9

Panellist Ratings vs. Panel Median

15
14

Grading Scores

13
12
11
10
9

Panelist Values

Product Result
(Median)

7
6
1

LC Quality Assurance
7

10

11

Samples

12

13

14

15

16

17

18

19

20

21

QA of Sensory Evaluation
Results

Internally designed to measure quality and

consistency of the sensory evaluation
System
assessments
Panel
Designed to measure:
System performance
Panellist
Panel Performance
Panellist Performance
Typical measures: repeatability, reproducibility,
bias, defects identification, etc.

LC Quality Assurance

Internal Proficiency Testing

Objective:

Assessment of results repeatability & reproducibility

Product Categ.: Ontario Wines

Methodology:

Assessment of panel results for three Ontario wines presented as blind duplicates.
Wines are of sound quality and similar quality levels.

Sensory Results
Average
SD

Sample 1

Sample 2

Sample 3

Replicate 1

Replicate 2

Replicate 1

Replicate 2

Replicate 1

Replicate 2

13.76
1.24

13.80
0.57

13.86
0.36

13.56
1.02

13.98
0.69

13.64
1.30

LC Quality Assurance

Internal Proficiency Testing

Repeatability
Repeatability Results: p-Values * (2-tailed test, 95 % Confidence Level)
Sample 1 Sample 2 Sample 3
1. Panel level
No significant difference between replicate results
0.8711
0.1895
0.3044
2. Panel groups level: No significant difference between replicates
Group A:
0.4382
0.2809
0.0759
Group B:
0.4814
0.5232
0.5746
Group C:
0.7688
0.5137
0.2235
Group D:
0.9653
0.5291
0.0823
3. Panellist level:
Only one taster exibited significant variability of the results.
Source of variation: Sample position in the line-up in reference with the sensory profile of
adjacent samples.

LC Quality Assurance

* Interpretation of results:: If the p-Value is lower than 0.05, then there is a signifficant difference at a 95% CL

Internal Proficiency Testing

Reproducibility
Sample

Results, Median Value (reporting value)

Replicate 1

Replicate 2

Replicate 1

Replicate 2

Replicate 1

Replicate 2

Replicate 1

Replicate 2

Group A Group A Group B Group B Group C Group C Group D Group D Panel

Nov. 8/04 Nov. 8/04 Nov. 10/04 Nov. 10/04 Nov.29/04 Nov.29/04 Nov.17/04 Nov.17/04
Ave.
Sample 1
13.2
13.6
14.6
14
13.4
13.7
13.8
13.8
13.8
Sample 2
13.8
13.8
13.6
13.6
13.7
13.7
14
14.4
13.6
Sample 3
14.4
13.6
13.8
14.4
13.7
14.2
13.2
14
13.6
Average SD:

Panel
SD
0.39370
0.25981
0.40552
0.35301

No significant difference between reported results, i.e.

median values (p-Value: 0.649671) at a 95% confidence
level

LC Quality Assurance

Internal Proficiency Testing

Reproducibility
Sample
Sample 1
Sample 2
Sample 3

Results, Median Value (reporting value)

Grading QA Panel Group A Group B Group C Group D Grading
Sept. 16/04 Sept.13/04 Sept.15/04 Sept.20/04 Sept.22/04 Panel SD
Panel
Average* (Ref. Panel)
12.9
13.8
13.8
13.5
13.4
13.2
0.25000
14.4
14.2
14.2
14.4
14.8
14.2
0.28284
10.7
10.6
10.2
9.4
9.8
12
1.14746
Average: 0.56010

* Average of individual panellists' score.

No significant difference between panels' (i.e. all 5

panels) reported results (i.e. median value), at a 95%
confidence level (p-value = 0.99605)

LC Quality Assurance

Sensory Evaluation Grading

System
Summary

A superior method for performing sensory evaluations.

Eliminates many sources of bias.
Customized for use with all beverage alcohol products.
Flexibility (data collection, quantification & reporting).
Provides significant time savings in data management.
Provides tools to analyze panellist results to help identify
training opportunities.

LC Quality Assurance

Leading
the Sensory
Evaluation of
Beverage
Alcohol

LC Quality Assurance