

Amazing
Audits
Evaluating an audit program’s
performance with a standardized
approach and the Kano model
by Lance B. Coleman Sr.

In 50 Words A QUALITY AUDIT program could

Or Less be considered a service for an organiza-
• A process and form al-
lows organizations to tion that’s provided by the quality depart-
evaluate the performance
of their audit programs. ment. Because of this, the Kano model can
• The evaluation’s score is
applied to the Kano mod- be the ideal lens for analyzing the perfor-
el, determining whether
the program satisfies the
mance of an organization’s quality audit
organization’s needs. program.
• The process allows or-
ganizations to track their In the 1980s, Noriaki Kano created the
audit programs’ status
over time and the effec- model as a theory of how to align product
tiveness of implemented
improvements. development with customer needs. Kano
 AUDITING

focuses on customer service and the ben-

efits of delivering exceptional value to the
customer, whether internal or external,
through service that delights the customer.
The Kano model recognizes four states:
nonperforming (failure), basic must haves
(cost of entry into the marketplace), per-
forming (more is better) and exciting or
delightful service (surprises and delights
the customer).1

September 2015 • QP X39

 The model (see Figure 1) stresses the pursuit of de-
lightful service delivery as a driver for continuous im-
provement and business success. It also can be used as
a powerful visual for displaying progress in customer
satisfaction, and it reminds staff of the of the organiza-
tion’s ultimate goal—delivering outstanding (delight-
ful) performance.
When I looked at this model, I asked myself: “What
would be attributes of a delightful audit program?”
A delightful program would excel in monitoring
compliance, managing risk and driving continuous im-
provement while the program itself continues to im-
prove. To know whether you have an excellent audit
program, an assessment must take place, but complet-
ing a quantifiable, accurate and objective assessment
of an internal quality audit program is easier said than
done.
I started by creating an evaluation form (see Figure
2, p. 43) to produce a ranking that could be incorpo-
rated into the Kano model, which provides a needed
framework for evaluating the service of an internal au-
dit program.
Evaluation forms
The evaluation form assesses critical areas of an audit
program, and it weights audit planning as the largest
component, allotting it 40% of the base score. Planning
is a critical element in any endeavor, ensuring all as-
pects of an organization that must be monitored are
consistently reviewed, and the evaluation form awards
points for conducting the audits as scheduled. After all,
Kano model / FIGURE 1
Satisfaction
+ gives the program credit for acting on opportunities for
Delighters
(excitement factors)
Primary satisfiers
Service Service
not fully
functional functional
Must-be
(basic requirements)
−
Dissatisfaction
40 QP • www.qualityprogress.com
having the best plan in the world is worthless if that
plan isn’t properly implemented.
The ability to support corporate and site goals is
an often-overlooked segment of an internal audit pro-
gram. It also should be seamlessly integrated into a
risk management program and play an integral role.
The next area of the form addressed reporting and
record keeping. For data to become information, it
must be used, which was why data analysis was added
to this area. Reporting, records and data analysis ac-
count for another 30% of the base score.
The reporting strategy for audit results was ad-
dressed to ensure critical and major concerns were im-
mediately reported to the appropriate parties. Safety,
legal and functional issues are examples of findings
that might be reported to a wider group of managers.
The final 30% of the score relates to having adequate
resources, organizational structure, variability and
breadth of the audit program’s impact. Some questions
that determine whether an audit program has the nec-
essary resources are:
• Are there enough auditors to complete the number
of comprehensive audits required within the given
timeframe?
• Do auditors have the experience, skills and educa-
tion to evaluate the areas they’re auditing?
• Is the required infrastructure—such as office space,
computers and supplies—needed to manage the
program in place?
• Is there a budget for travel if off-site audits are re-
quired?
• Are there authority and management support to con-
duct audits?
After a base score has been attained, you will evalu-
ate how the audit program provides value beyond
verifying compliance and fulfilling requirements. This
improvement (see Table 1). If those opportunities had
potential financial savings attached to them, the audit
program received additional points.
The next step is called the reality check (see Table
2). Each major finding against the audit program found
during external audits, including findings previously
identified by the internal audit program that did not
have successful corrective action implemented, sub-
tracts points from the base score. Finally, there are
point deductions for findings related to issues not pre-
viously noted by the internal audit program and repeat
 AUDITING

findings from the previous year. An organization also

should include repeat findings that resulted from an
Points added for value-add
external audit.
findings / TABLE 1
These points are added or subtracted from the base
Value add
score (see Tables 1 and 2) using a five-point rating
Points Description
scale:
+5 Observation that lead directly to $1,000+ lean savings.
• Five is very good or the best possible.
• Four is above average. +2 Observation that lead directly to $1-$1,000 lean savings.

• Three is minimally acceptable.
• Two needs improvement.
• One is very poor.
• Zero is nonexistent or almost nonexistent.
These definitions should be used to help score each
category. The sum of these scores is converted to a
100% scale where 80% or greater meets the goal, 60% is
minimally passing and 90% or greater (a minimum rat-
ing of three out of five in every category) is considered
excellent—or delightful on the Kano model.
This assessment model allows an organization to
define each rating according to its guidelines. Simi-
larly, the users decide what is adequate or more-than-
adequate training for auditors, including the level of
resources needed to carry out the mandates of their
programs. After an organization defines its ratings,
the model should be consistently applied to provide
results that can be analyzed and used as input to the
continuous improvement program.
The audit cycle
An audit program manager (APM) develops plans
based on organizational, site and audit program goals,
which include various aspects of compliance, improve-
ment and risk management. The APM may receive indi-
vidual audit plans for review and approval, but this de-
pends on an organization’s policy and the experience
of the lead auditor.
The APM drives the audit’s execution phase by pro-
viding policies, procedures, resources and credential-
ing requirements, and he or she will include this in the
auditors’ assignments. APMs receive reported audit
results from lead auditors, and they analyze those find-
ings and report them to site management.
Responses to the audit are included as input in de-
velopment of future plans. This entire cycle (see Online
Figure 1, which can be found on this article’s webpage
at www.qualityprogress.com) is repeated with momen-
tum from ongoing revisions of goals and objectives,
including analysis of data provided by the audit results.
+0.5 Opportunity for improvement that is implemented by area
management.
Points deducted based on results
of external audits / TABLE 2
Reality check
Points Description
−5 Each major finding against audit program from outside audit.
−3 Each major finding from outside audit that was previously
identified internally.
−2 Each major finding against quality management system from
outside audit.
−1 Each minor finding against audit program from outside audit.
−1 Each repeat finding from previous year.
After my audit program went through this rating
process, I looked at the score and found the initial rat-
ing reflected my first impression of the program when
I arrived at the site. The improvements I made during
my first year resulted in a better score, and it supported
my anecdotal and subjective opinion of the program’s
status at that time.
The evaluation form’s elements were chosen be-
cause they’re as objective as possible and likely to be
consistent on a yearly basis.
Other factors
Note that there are other important measures of an
audit program’s success not included in this model.
This is because there are factors—beyond how robust
an audit program is—that might be involved in its im-
provement: for example, a yearly reduction in custom-
er audit findings.
Other possible reasons for improvement that are
unrelated to an audit program’s performance include:
different auditors, stricter clients, improvement in the

September 2015 • QP 41
 quality management system or luck of the draw (an au- components to meet your organization’s needs, and use
dit only takes a sample after all). it as an assessment of your assessment program. QP
Even other factors, whether they’re minor or major, REFERENCE
can contribute to a reduction of internal audit findings, 1. Kano Model Tutorial, ASQ.org, http://tinyurl.com/kanomodelasq.

and it’s difficult to consistently quantify monetary sav-

ings from findings.
These key performance indicators should, however,
improve yearly as your audit program’s rating and the
program itself improve. Learn how to use this evalua-
tion tool, and read the sidebar “Case In Point,” which
illustrates an example of an audit program before and
after improvements were made using this model. You
can use the evaluation form in Figure 2 as a guide. Ad-
just the ratings’ weights and add, change or remove
LANCE B. COLEMAN SR. is a quality engineer and lean
program coordinator at The Tech Group in Tempe,
AZ. Coleman holds a degree in electrical engineer-
ing technology from the Southern Polytechnical
University in Marietta, GA. A senior member of ASQ,
Coleman is an ASQ-certified quality engineer, Six
Sigma Green Belt, quality auditor, biomedical auditor
and an Exemplar Global quality management system
principal auditor. Coleman is the author of Advanced Quality Auditing: An
Auditor’s Review of Risk Management, Lean Improvement and Data Analysis
(ASQ Quality Press, 2015). He is an instructor for ASQ’s quality auditor
certification refresher course and is the newsletter editor for ASQ’s Lean
Enterprise and Audit Divisions.

CASE IN POINT
Here is an example of how to assess an audit program based
on existing records. Newly certified organization ABC was hav-
ing trouble getting its internal audit program off the ground.
A consultant was hired to improve the internal quality audit
program after the organization received findings (one major and
one minor) against it in each of its two customer audits during
the preceding year. The auditors had received formal training,
and there were no dedicated auditors; auditing was a second-
ary job function.
After reviewing ABC’s records, the consultant noted the fol-
lowing:
1. The audit schedule for the previous year was not completed
until June.
2. Only seven of 17 scheduled audits were conducted.
3. There was objective evidence that only one of two opportuni-
ties cited was followed up on.
4. The status of scheduled audits was not tracked.
5. Audit forms were completed inconsistently from person to
person.
6. Auditor meetings were not held.
7. Audit results were not regularly reported.
8. Training records were available only for four of six current
auditors.
9. Audit forms were poorly designed and did not logically flow.
10. Audit procedures were unclear and insufficiently detailed.
The consultant interviewed the existing auditors, and many
expressed dissatisfaction with their levels of training. All
confessed to occasional confusion about how to classify audit
findings, how to complete various audit working papers and
when to pursue formal corrective action. They said it was due to
unclear work instructions and poorly designed forms. Some were
concerned with having too many audits to do along with their
regular workloads.
Before beginning a root cause and corrective action pro-
cess analysis, the consultant rated ABC’s internal quality audit
program to provide an objective baseline for comparison after
improvements were implemented. Figure 3 (p. 44) shows the
results of the assessment. A score of 60 is minimally acceptable,
80 or better is the goal and a score of 90 or higher is delightful.
With a rating of 23% (based on the information from ABC’s
records), ABC’s audit program was in the dissatisfaction quadrant
of the Kano model (see Online Figure 2, which can be found on
this article’s webpage at www.qualityprogress.com).
The next year, the following changes were implemented:
• Two new auditors were trained, and one worked on off shifts.
• Two of the existing auditors received refresher training.
• Portions of two audits were conducted during off shifts.
• The standard operating procedure for audits was completely
rewritten.
• Audit forms were revised, and auditors were trained to ad-
dress previously cited deficiencies.
These changes led to the following results from subsequent
customer audits. There were no major audit findings against the
quality management system, and there wasn’t any nonconfor-
mance issued against the audit program itself. Figure 4 (p. 45)
shows the results of the following year’s audit program evaluation.
The improvements made to the audit program were substan-
tial, and as expected, they were reflected in the greatly improved
score. The new rating was more than three times greater than the
original score—well above minimally acceptable requirements
and within the margin of error for attaining the goal of 80%.
The score was in the must-be quadrant of the Kano model
(see Online Figure 2). There were still improvements that could
be made, and some areas’ low scores could easily be used as
a roadmap for improvement and delightful performance in the
future. —L.C.

42 QP • www.qualityprogress.com
 AUDITING

Audit system evaluation form / FIGURE 2

Audit system evaluation

Date: 

  Planning (40%)
___  5%—All audits are scheduled that should be scheduled (per requirements of the standard, regulation or internal policies).
___  5%—All scheduled audits were conducted.
___  5%—Percentage that were on time.
___  5%—All nonscheduled audits (not on the annual schedule, but scheduled to address a concern that was identified)
were conducted.
___  5%—Annual audit plan was approved by site management.
___  5%—Audit program is tied to quality management system (QMS) evaluation metrics and corporate goals.1
___  5%—Individual audit plans were approved by lead auditor or quality management with input from process owners.
___  5%—Audit program is well integrated within the risk management program.

  Reporting, records and analysis (30%)

Reporting—10%
___  Reported to site-level management.
___  Reported to quality manager.
Data analysis—10%
___  Reported data regularly analyzed and acted on.
Records—10%
___  Records maintained (audit reports).
___  Records maintained (training records).
___  Records maintained (quarterly or annual summary reports).

  Implementation and results (30%)

___  5%—Do we audit across all operating shifts?
___  5%—Is there an appropriate level of auditor training (requirements + actual)?
___  5%—Are an appropriate amount of audits dedicated to improvement activities?2
___  5%—Is a documented and structured method for evaluating the validity and classification of findings?
___  5%—Are there adequate resources?
___  5%—Are varying types of audits used, such as process, product, trace, system or element?

  Subtotal = potentially 100%

+   Value add percentage
–   Reality check percentage
 Total

Notes
1. ISO 9001:2008 and ISO 13485:2003 requires an audit program to monitor the effectiveness of a QMS, but it isn’t explained how you
know whether the QMS is effective. It’s not enough to simply state that a lack of findings demonstrates that the QMS is effective.
Metrics should be developed to assess the QMS, and the audit program should monitor those metrics in addition to compliance to
policies and procedures.
2. Improvement auditing is assessed against criteria that could be driven by compliance or a desired future state. It also could evaluate an
area for muda (nonvalue-add wastes) or monitor project results against predetermined milestones.

September 2015 • QP 43
 Organization ABC’s pre-improvement evaluation / FIGURE 3

Audit system evaluation

Date: mm/dd/yyyy

17%  Planning (40%)  17 of possible 40 total rating x 40% = (17/40) x 0.4 = 0.17 = 17%
  3   5%—All audits are scheduled that should be scheduled (per requirements of the standard, regulation or internal policies). 
Yes, but were points deducted because the schedule wasn’t released until June.
  1    5%—All scheduled audits were conducted.  Only seven of 17 scheduled audits were conducted.
  0    5%—Percentage that were on time.
  5    5%—All nonscheduled audits (not on the annual schedule and scheduled to address a concern that was identified)
were conducted.  There were no nonscheduled audits (full credit).
  3    5%—Annual audit plan was approved by site management.  It was developed by the quality assurance manager.
  0    5%—Audit program is tied to quality management system (QMS) evaluation metrics and corporate goals. Not done.
  3    5%—Individual audit plans were approved by lead auditor or quality management with input from process owners. 
No input was provided from process owner.
  2    5%—Audit program is well integrated within the risk management program.  It’s informally tied to the risk management program.

13%  Reporting, records and analysis (30%)  13 of possible 30 total rating x 30% = (13/30) x 0.3 = 0.13 = 13%
Reporting—10%
  1    Reported to site-level management.  Only the number of audits conducted were reported during the management review.
  5    Reported to quality manager.  Results were reported to the process owner, and the quality manager signs each report for closure.
Data analysis—10%
  0    Reported data regularly analyzed and acted on.
Records—10%
  5    Records maintained (audit reports).  Hard and electronic copies of audit reports and schedule were maintained.
  2    Records maintained (training records).  Some training records were missing.
  0    Records maintained (quarterly/annual summary reports).  Not done.

10%  Implementation and results (30%)  10 of possible 30 total rating x 30% = (10/30) x 0.3 = 0.10 = 10%

  0    5%—Do we audit across all operating shifts?  Auditors work and audit only during the day shift.
  2   5%—Is there an appropriate level of auditor training (requirements + actual)? 
Auditors were trained, but requirements were not specified.
  0   5%—Are an appropriate amount of audits dedicated to improvement activities? 
Only compliance-based audits were conducted.
  3    5%—Is a documented and structured method for evaluating the validity and classification of findings?  They could be clearer.
  3    5%—Are there adequate resources?  There were adequate resources to implement the program.
  2    5%—Are varying types of audits used, such as process, product, trace, system or element?  Only element audits were done.

40%  Subtotal = potentially 100%

+   1%  Value add percentage  One point was added to score for an identified opportunity that was implemented.
– 18%  Reality check percentage Five points were deducted for one major audit program finding, -12 for six major QMS findings,
and -1 for one minor audit program finding.
23% Total

44 QP • www.qualityprogress.com
 AUDITING

Organization ABC’s post-improvement evaluation / FIGURE 4

Audit system evaluation

Date: mm/dd/yyyy

31%  Planning (40%)  31 of possible 40 total rating x 40% = (31/40) x 0.4 = 0.31 = 31%

  5   5%—All audits are scheduled that should be scheduled (per requirements of the standard, regulation or internal policies).
  5    5%—All scheduled audits were conducted.
  4    5%—Percentage that were on time.  80% of audits conducted on time; including overflow from previous year.
  5    5%—All nonscheduled audits (not on the annual schedule and scheduled to address a concern that was identified)
were conducted.  There were no nonscheduled audits (full credit).
  4    5%—Annual audit plan was approved by site management.  Attachment in standard operating procedures was approved by
site management.
  1    5%—Audit program is tied to quality management system (QMS) evaluation metrics and corporate goals.
Improvement in this area was initiated.
  4    5%—Individual audit plans were approved by lead auditor or quality management with input from process owners. 
Now they are input from process owner.
  3    5%—Audit program is well integrated within the risk management program.  Findings are now classified by risk.

26%  Reporting, records and analysis (30%)  26 of possible 30 total rating x 30% = (26/30) x 0.3 = 0.26 = 26%

Reporting—10%
  4    Reported to site-level management.  They provided quarterly and annual summary reports, and input to management review.
  5    Reported to quality manager. Results were reported to process owner, and quality manager signed each report for closure.
Data analysis—10%
  2    Reported data regularly analyzed and acted on.  Improvement in this area was initiated.
Records—10%
  5    Records maintained (audit reports).  Hard and electronic copies of audit reports and schedule were maintained.
  5    Records maintained (training records).  All training records were current and available.
  5    Records maintained (quarterly/annual summary reports). 

16%  Implementation and results (30%)  16 out of possible 30 total rating x 30% = (16/30) x 0.3 = 0.16 = 16%

  2    5%—Do we audit across all operating shifts?  Parts of two audits conducted on off shift.
  4   5%—Is there an appropriate level of auditor training (requirements + actual)? 
  1   5%—Are an appropriate amount of audits dedicated to improvement activities?  Improvement in this area was initiated.
 3.5   5%—Is a documented and structured method for evaluating validity and classification of findings? 
 3.5   5%—Are there adequate resources?  There were adequate resources to implement the program.
  2    5%—Are varying types of audits used, such as process, product, trace, system or element?  Only element audits were done.

73%  Subtotal = potentially 100%

+   6%  Value add percentage  Six points were added to the score for identified opportunities that were implemented.
–   0%  Reality check percentage
79% Total

September 2015 • QP 45