Service Quality Assessment in a Digital Library Environment

4th International JISC/CNI Conference

Joseph Boykin, Fred Heath, Duane Webster
Edinburgh, Scotland June 27, 2002

Overview of Discussions
1. New Models for Understanding and Describing Library Success 2. ARLs New Measures Initiative 3. LibQUAL+ Project Development 4. Experience with LibQUAL+ 5. Conclusions and Next Steps

The Association of Research Libraries

Mission: Shaping and influencing forces affecting the future of research libraries in the process of scholarly communication. 123 major research libraries in North 4% of the higher education institutions providing 40% of the information resources. 3 million students and faculty served.

Members: America. Ratios: Users:

www.arl.org

Expenditures: $2.35 billion annually, $727 million for acquisitions of which 9% is invested in access to electronic resources.
ASSOCIATION OF RESEARCH LIBRARIES

The Problem of Assessment in Research Libraries

ARL Membership Criteria Index variables emphasize inputs, primarily expenditures To rise in the ARL Index it is only necessary to spend more No demonstrable relationship between expenditures and service quality The lack of metrics describing performance

Systems Model for Organization Design

ENVIRONMENT
FUTURE ENVIRONMENT

MISSION
INPUTS WO RK PRO CE SSES RESOURCES HUMAN AS PE CT S FEEDBACK FEEDBACK OUTPUTS

SUPPLIERS

CUSTOMERS
OUTCOMES

&

VISION

STRUCTURE / SYSTEMS

PEOPLE / SKILLS CULTURE / CLIMATE

FROM : Association of R ese arch Libraries, Office of Leadership and Management Services, Washington DC

STAKEHOLDERS

ARL New Measures Initiative

Collaboration among member leaders with strong interest in this area Specific projects developed with different models for exploration Intent to make resulting tools and methodologies available to full membership and wider community

ARL New Measures Projects

Demonstration project for service effectiveness measures (LibQUAL+) Project to define usage measures for electronic information resources (includes institutional outcomes) Investigation of role libraries play in support of the research process Investigation of role libraries can play in campus learning outcomes activities Identification of cost-drivers and development of cost-benefit studies

Assessment
The difficulty lies in trying to find a single model or set of simple indicators that can be used by different institutions, and that will compare something across large groups that is by definition only locally applicable i.e., how well a library meets the needs of its institution. Librarians have either made do with oversimplified national data or have undertaken customized local evaluations of effectiveness, but there has not been devised an effective way to link the two Sarah Pritchard, Library Trends, 1996

LibQUAL+ Description
LibQUAL+TM is a research and development project undertaken to define and measure library service quality across institutions and to create useful quality-assessment tools for local planning.

The Purpose of Our Research

To fill a knowledge void in modeling the dimensions of library service quality from a user perspective Based upon the model, to develop a webdelivered, effective total market survey instrument equivalent for service quality assessment in academic libraries Using the derived instrument to recommend a process for an ongoing program of comparative outcome measurement for academic libraries

LibQUAL+ Project Goals

1. Development of web-based tools for assessing library service quality 2. Development of mechanisms and protocols for evaluating libraries 3. Identification of best practices in providing library service 4. Establishment of a library service quality assessment program at ARL

Project Resources
LibQUAL+TM is an ARL/Texas A&M University joint effort. The project is supported in part by a 3-year grant from the U.S. Department of Educations Fund for the Improvement of Post-Secondary Education (FIPSE).

TM LibQUAL+ Team

ARL Duane Webster Martha Kyrillidou Kaylyn Hipps Julia Blixrud Jonathan Sousa Consuella Waller

TAMU Fred Heath Colleen Cook Bruce Thompson Yvonna Lincoln Trey Thompson Julie Guidry

LibQUAL+TM Project History

9/99 - ARL launches New Measures Initiative which includes the study of service effectiveness known as SERVQUAL spearheaded by Texas A&M University. 1/00 - Initial 12 institutions begin the SERVQUAL study led by the Texas A&M team. 7/00 - LibQUAL+ as a distinct library-based assessment tool is presented. 9/00 - ARL and Texas A&M awarded a FIPSE grant to fund further development of the LibQUAL+ project. 10/00 - The ARL symposium, New Culture of Assessment in Academic Libraries Measuring Service Quality attracts a group of 170 people. 1/01 - Representatives from 43 research and university libraries participating in the Spring 2001 Implementation meet in Washington, DC during ALA midwinter. 6/01 - National Science Foundation awards grant to ARL and Texas A&M to adapt LibQUAL+ for NSDL

LibQUAL+ Contribution
Grounded questions yield data of sufficient granularity to be of local use Normative data across cohort group at first and higher order levels Surfaces Best Practices Web approach makes little demand of local resources while compiling robust dataset

Relationships: perceptions, service quality and satisfaction

.only customers judge quality; all other judgments are essentially irrelevant Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.

LibQUAL+TM Participants
Year 3
Year 2 43 Participants Year 1 12 Participants 170 Participants

Spring 2000

Spring 2001

Spring 2002

For More Information about Participants: Visit the LibQUAL+ web site.

SURVEY INSTRUMENT

Dimensions of Library Service Quality

Affect of Service
Empathy Responsiveness Assurance Reliability

Library Service Quality

Personal Control
Ease of Navigation Convenience Modern Equipment

Information Access
Content/Scope Timeliness Convenience

Library as Place
Utilitarian space Symbol Refuge

LibQUAL+ Core Questions Y1

_____________________________________________________________________________ Factor_ ______ _ No. I II III IV Item Core _____________________________________________________________________________ 32 .84947 .12848 .24465 .13335 1 Willingness to help users 33 .80847 .13662 .25348 .14147 1 Giving users individual attention 7 .80757 .17881 .12781 .21125 1 Employees deal with users caring fashion 50 .79273 .19288 .18847 .12497 1 Employees who are consistently courteous 31 .77262 .16358 .26461 .20061 1 Employees have knowledge answer questions 5 .74072 .14754 .18453 .29624 1 Employees understand needs of users 3 .74052 .15102 .17296 .20793 1 Readiness to respond to users' questions 18 .71718 .19757 .18289 .26766 1 Employees who instill confidence in users 43 .62487 .22402 .29970 .28256 0 Dependability handling service problems 20 2 19 25 41 37 28 14 45 17 29 11 39 16 9 8 .16556 .17739 .22362 .16013 .20398 .22528 .19602 .33339 .30467 .35390 .30136 .13494 .14894 .29445 .27782 .22850 .87679 .83172 .83147 .80492 .80204 .12353 .09611 .16156 .23784 .18467 .21018 .23183 .23743 .19831 .05333 .18484 .11430 .08498 .14705 .18894 .17599 .78405 .75780 .60389 .59090 .55690 .55341 .18868 .29367 .22384 .16331 .13137 .16236 .13901 .22566 .16628 .20255 .15466 .13173 .31109 .28919 .41864 .38474 .73636 .60350 .60107 .57866 .56343 2 A haven for quiet and solitude 2 A meditative place 2 A contemplative environment 2 Space that facilitates quiet study 2 A place for reflection and creativity * website enabling me locate info on my own * elec resources accessible home or office * access tools allow me find on my own 3 Modern equip me easily access info I need * info easily accessible for independent use 4 Convenient access to library collections 3 Comprehensive print collections 3 Complete runs of journal titles 3 Interdisciplinary library needs addressed 4 Timely document delivery/interlibrary loan 0 Convenient business hours

________________________________________________________________

Affect of Service
Emerged as the dominant factor early in our work Absorbed several of the original SERVQUAL questions measuring Responsiveness, Assurance and Empathy In the current analysis also includes Reliability All in all: the Human Dimension of Service Quality

Library as Place
Transcends the SERVQUAL dimension of Tangibles to include the idea of the library as the campus center of intellectual activity As long as physical facilities are adequate, library as place may not be an issue

Personal Control
How users want to interact with the modern library Personal control of the information universe in general and web navigation in particular

Access to Information
Ubiquity of access: information delivered in the format, location and time of choice Comprehensive collections

Project Deliverables
Print and web-based results include:
Aggregate Summaries Demographics by Library Item Summaries Dimension Summaries A copy of the survey instrument Dimensions measured for survey implementation

Two Interpretation Frameworks

Score Norms Zone of Tolerance

Score Norms
Norm Conversion Tables facilitate the interpretation of observed scores using norms created for a large and representative sample. LibQUAL+TM norms have been created at both the individual and institutional level

Zone of Tolerance
The area between minimally acceptable and desired service quality ratings Perception ratings ideally fall within the Zone of Tolerance

ZONE OF TOLERANCE: DIMENSIONS

LibQUAL+ Fundamental Contributions to the Measurement of Effective Delivery of Library Services

Shift the focus of assessment Determine the degree to which information derived from local from mechanical expendituredata can be generalized, driven metrics to user-centered providing much needed best measures of quality practices information Re-ground gap theory for the library sector, especially academic libraries Demonstrate the efficacy of large-scale administration of user-centered assessment transparently across the web

LibQual+ 2000 and 2001 Results

LIBQUAL+ Category: Access to Information

2001 2002

Access to Information Timely document delivery/interlibrary loan Comprehensive print collections Interdisciplinary library needs being addressed

All 0.19 -0.67 0.24

All 0.5 -0.06 0.26

Change 0.31 0.61 0.02

Complete runs of journal titles Convenient Business Hours

-0.97 -0.03

-0.12 0.62

0.85 0.65

Positive Gaps 0.60 or higher Negative Gaps (regardless of value)

LIBQUAL+ Category: Affect of Service

Affect of Service Readiness to respond to users' questions Employees who understand the needs of their users Employees who deal with users in a caring fashion Employees who instill confidence in users Employees who have the knowledge to answer user questions W illingness to help users Giving users individual attention Dependability in handling users' service problems Employees who are consistently courteous 2001 All 0.54 0.36 0.74 0.76 0.44 0.61 0.76 0.40 0.63 2002 All Change 0.63 0.09 0.55 0.19 0.78 0.04 0.91 0.15 0.41 -0.03 1.00 0.39 0.89 0.13 0.50 0.10 0.74 0.11

Positive Gaps 0.60 or higher Negative Gaps (regardless of value)

LIBQUAL+ Category: Library as Place

Library as Place A contemplative environment A haven for quiet and solitude Space that facilitates quiet study A comfortable and inviting location A place for reflection and creativity

2001 All 0.12 0.07 0.16 0.64 0.38

2002 All Change 0.50 0.38 0.52 0.45 0.39 0.23 0.65 -0.01 0.54 0.16

Positive Gaps 0.60 or higher Negative Gaps (regardless of value)

LIBQUAL+ Category: Personal Control

Personal Control Easy-to-use access tools that allow me to find things on my own Making information easily accessible for independent use Making electronic resources accessible from my home or office A library website enabling me to locate information on my own Modern equipment that lets me easily access the information I need Convenient access to library collections

2001 All 0.28 0.15 -0.05 0.14 -0.11 0.41

2002 All Change 0.26 -0.02 0.31 0.16 0.19 0.24 0.12 -0.02 0.34 0.45 0.32 -0.11

Positive Gaps 0.60 or higher Negative Gaps (regardless of value)

NSF Grant
Assess service quality in digital libraries 3 year period Adopt LibQUAL+ instrument for use in the Science, Math, Engineering and Technology Education Digital Library community (NSDL)

NDSL LibQUAL+ Goals

Define dimensions of digital library service quality from the users perspectives Develop tool for measuring user perceptions and expectations of digital library service quality across NSDL digital library contexts Identify digital library best practices that permit generalizations across operations and development platforms

NDSL LibQUAL+ Activities

120-200 qualitative interviews to contribute to identifying dimensions of digital library service quality Test and refine dimensions of digital library service quality and self-sufficiency through development of total market survey Implement survey across variety of organizational and digital library implementations

LibQUAL+TM Related Documents

LibQUAL+TM Web Site
http://www.arl.org/libqual/

LibQUAL+TM Bibliography
http://www.coe.tamu.edu/~bthompson/servqbib

Survey Participants Procedures Manual

http://www.arl.org/libqual/procedure/lqmanual2.pd f

 LibQUAL+ Related

Documents
LibQUAL+ Web Site
http://www.arl.org/libqual/

LibQUAL+ Bibliography
http://www.coe.tamu.edu/~bthompson/servqbib
woof