Bominique Santoielli

S72: Evaluation of Libiaiy anu Infoimation Seivices anu Systems

Teim Papei
}uly 24, 2u1S

Bigital Libiaiy Evaluation
As technology auvances, moie infoimation iesouices aie being uigitizeu anu aie
incieasingly boin uigital, anu uigital libiaiies aie being cieateu to house these iesouices.
These uigital libiaiies can be inuepenuent of oi an extension of tiauitional libiaiies.
Bespite numeious challenges, ieseaicheis aie making effoits to select ciiteiia that can be
useu foi evaluation of uigital libiaiies as well as methouologies foi collecting anu analyzing
that uata. While tiauitional libiaiy evaluation techniques can be applieu to uigital libiaiies,
they must often be auapteu to accommouate the technological featuies of a uigital libiaiy.
Cuiient evaluation ciiteiia anu methous incoipoiate techniques fiom tiauitional libiaiies,
systems, infoimation ietiieval, anu human-computei inteiaction in an attempt to covei all
aspects of evaluation, but fail to incluue ciiteiia foi aspects unique to uigital libiaiies. 0sei
satisfaction, an impoitant consiueiation foi all types of libiaiies, piesents valuable
evaluation ciiteiia foi uigital libiaiies. As a iesult, usability has become a fiequently useu
evaluation measuie as it maiiies effective use of a system anu usei satisfaction. Bue to
numeious challenges, a stanuaiu evaluation mouel foi uigital libiaiies has yet to be
cieateu.
Pait of the challenge of ueveloping an evaluation fiamewoik is the lack of a
uefinition foi a uigital libiaiy. Befinitions of a uigital libiaiy vaiy baseu on uiffeient
people's peispectives. "To libiaiians, a uigital libiaiy is anothei foim of a physical libiaiy;
to computei scientists, a uigital libiaiy is a uistiibuteu text-baseu infoimation system oi a
netwoikeu multimeuia infoimation system; to enu useis, uigital libiaiies aie similai to the
woilu wiue web (www) with impiovements in peifoimance, oiganization, functionality,
anu usability" (Xie 1S46). Bigital libiaiies aie laigely iecognizeu as an oiganizeu collection
of uigital woiks anu the means to access them, but vaiying peiceptions incluue auuitional
iuentifieis anu bounuaiies aie uifficult to uefine. These uiffeient uefinitions iesult in
uiffeiing piioiities anu theiefoie, uiffeiing appioaches anu evaluation ciiteiia. Saiacevic
(2uuS) explains "uigital libiaiies aie complex social, institutional anu technical systems. No
evaluation can possibly auuiess all of these aspects togethei. Thus, uiffeient appioaches
have been useu foi uiffeient evaluation goals" (S). Be iuentifies the most commonly useu
appioaches cuiiently useu foi evaluation, systems-centeieu appioach, human-centeieu
appioach, anu usability-centeieu appioach, anu seveial less fiequently stuuieu appioaches.
Cuiient uigital libiaiy stuuies evaluate iepiesentations of infoimation, tools, seivices, usei
behavioi, anu pievious evaluation methous uepenuing on the peispective of the ieseaichei
anu theii chosen appioach.
Befoie ueciuing what to evaluate oi how to uo so, the question of why to evaluate, oi
the objectives, must be answeieu. Saiacevic (2uuu) explains, "The answei to the fiist
question as to why to evaluate shoulu seive as a base foi selection of an appiopiiate
appioach" (SS8). Beteimining the goals of the evaluation piocess can be a uifficult piocess
uue to fiequent oveilap, such as whethei a uigital libiaiy is meeting a usei's neeus anu how
accessible it is. Saiacevic (2uuu) uiviues objectives into seven geneial classes: social,
institutional, inuiviuual, inteiface, engineeiing, piocessing, anu content (S6S-4). Ciiteiia
can then be ueteimineu foi each objective. "While the objectives may be explicitly stateu oi
implicitly ueiiveu oi assumeu, they have to be ieflecteu in an evaluation. Evaluation is not
one fixeu thing. Foi the same system, evaluation can be uone on uiffeient levels, in ielation
to uiffeient choices of objectives, using a vaiiety of methous, anu it can be oiienteu towaiu
uiffeient goals anu auuiences" (Saiacevic & Covi, S). They iecommenu that an evaluation
specifies constiuct, context, ciiteiia, measuies, anu methouology. Evaluation must incluue
cleai selections anu uecisions in each categoiy befoie piogiessing to the next.
Evaluation appioaches anu metiics foi uigital libiaiies aie cuiiently being ueiiveu
fiom ciiteiia useu in tiauitional libiaiies. Bigital libiaiies aie often seen as ielateu to oi
extensions of physical libiaiies because they peifoim many similai functions anu seivices.
Existing uata collection methous anu metiics fiom tiauitional libiaiies such as ciiculation,
gate counts, collection size, pation satisfaction anu cost pei use can be applieu to uigital
libiaiies. 0thei impoitant vaiiables incluue "accessibility, availability, seaichability, anu
usability" (Xie 1S48). Saiacevic (2uuu) outlines the following tiauitional libiaiy ciiteiia
anu suggests they can anu shoulu be applieu to uigital libiaiies: collection, infoimation, use,
anu stanuaius (S66). 0sage is often useu as a ciiteiion foi evaluation as stuuying patteins
pioviues helpful uata foi making uecisions, such as which mateiials aie fiequently
iequesteu anu cost pei use. As with tiauitional libiaiies, efficiency anu financial metiics
aie valuable in the assessment of uigital libiaiies.
Bowevei, uigital libiaiies uiffei fiom tiauitional libiaiies in many aspects. "Bigital
libiaiies maiiy the missions, techniques, anu cultuies of physical libiaiies with the
capabilities anu cultuies of computing anu telecommunications. Evaluating uigital libiaiies
is a bit like juuging how successful is a maiiiage. Nuch uepenus on how successful the
paitneis aie as inuiviuuals as well as the emeigent conuitions maue possible by the union"
(Naichionini Su4). Evaluation is best appioacheu by giouping the ciiteiia into similai
categoiies such as tiauitional libiaiy, technology, anu the oveilap between the two. Fuhi et
al. (2uu1) appioach uigital libiaiy evaluation by uiviuing it into foui uimensions: useis,
usage, collection, anu technology, which seive as the piinciple ieseaich aieas foi
assessment. While many functions can be evaluateu similaily to tiauitional libiaiies,
technology is one of the ciucial uiffeiences between the evaluation of tiauitional libiaiies
anu uigital ones. Technology iequiies that ceitain functions of a uigital libiaiy, such as
access, be evaluateu uiffeiently than its physical counteipait. 0sei stuuies must incluue
auuitional evaluation ciiteiia, as ability to navigate systems anu othei technical capabilities
have an effect on theii use of anu satisfaction with uigital libiaiies.
The technology-ielateu aspects of uigital libiaiies iequiie auuitional ciiteiia when
piepaiing to evaluate them. As tiauitional libiaiy ciiteiia aie mouifieu foi uigital libiaiies,
tiauitional systems ciiteiia can be applieu to uigital libiaiies. Fuhi et al. (2uu1) subuiviue
the technology uimension into foui aieas, usei technology, infoimation access, systems
stiuctuie, anu uocument technology, each with its own suggesteu topics foi evaluation.
Tiauitional IR ciiteiia foi the evaluation of systems anu tiauitional human-computei
inteiaction ciiteiia can be incoipoiateu in the assessment of uigital libiaiies. A systems
appioach offeis a vaiiety of ciiteiia foi uigital libiaiy evaluation anu is one of the most
commonly useu appioaches. "A human-maue system, such as a uigital libiaiy, has an
auueu aspect: it has ceitain objective(s). The elements, oi the components, inteiact to
peifoim ceitain functions, oi piocesses, to achieve given objectives" (Saiacevic & Covi S).
They suggest that with a systems appioach the uigital libiaiy system's peifoimance can be
evaluateu in teims of effectiveness, efficiency, anu the combineu cost-effectiveness.
Netiics useu to evaluate infoimation ietiieval aie often incluueu in this type of appioach.
"Aspects of technology peifoimance, piocessalgoiithm peifoimance, anu oveiall system
such as iesponse time, eiioi iates, maintainability, anu inteiopeiability have been useu in
vaiious stuuies (Saiacevic 2uuS 7). Netiics such as iecall anu piecision aie typically useu,
anu a stanuaiu set of pioceuuies that incluues test questions anu pooleu ielevance
juugments aie useu to ensuie compaiability acioss systems" (Naichionini S12).
Biffeientiating between evaluating infoimation ietiieval anu evaluating a uigital libiaiy
can piesent a challenge. Bepenuing on the peispective of the ieseaichei, the chosen
ciiteiia foi a systems-baseu appioach to evaluation may not pioviue enough infoimation
foi stakeholueis to make uecisions iegaiuing the uigital libiaiy as a whole, but this uata
enables uevelopeis to make changes to the uigital libiaiy that will impiove access anu the
usei expeiience.
With uigital libiaiies still in uevelopment stages, theie aie many unansweieu
questions about theii use anu the usei. As with a tiauitional libiaiy, knowing the usei is
essential in satisfying theii infoimation neeus. A human-centeieu appioach to uigital
libiaiy evaluation consiueis the ways useis aie using uigital libiaiies anu the puipose of
theii seaich. "The uses anu the usei aie intimately connecteu with the foui basic questions
that can be askeu about any maiket," namely who, what, wheie, anu why (Fuhi 2uu1).
0thei questions incluue "whethei useis use them, how they use them, anu what facilitates
anu hinueis theii access of infoimation in these uigital libiaiies" (Xie 1S47). A human-
centeieu appioach must account foi a wiue vaiiety of useis anu seives many uiffeient
libiaiy stakeholueis. This type of appioach "may taiget uiffeient goals ianging fiom
incieaseu leaining anu impioveu ieseaich to impioveu uissemination to bottom line
piofits" (Saiacevic & Covi 6). Bemogiaphic infoimation coupleu with usage uata can be
inteipieteu to pioviue extiemely valuable qualitative uata that can be useu to infoim
uecisions about the uigital libiaiy in teims of system, collection, anu seivices as well as its
ielation to a tiauitional libiaiy.
Because the usei's peiception of the uigital libiaiy is consiueiable factoi in the
evaluation piocess, ieseaicheis stuuy useis anu usei behavioi to ueteimine what qualities
of a uigital libiaiy aie impoitant to them. Foi example, }eng's 2uuS stuuy soliciteu the
usei's peiception of ciiteiia such as "ease of use, oiganization of infoimation, teiminology,
attiactiveness, anu mistake iecoveiy" (Xie 1SSu). Liteiatuie on uigital libiaiy usei
behavioi also coveis topics such as "usage patteins of seivice logs, peiception of quality in
uigital libiaiies by euucatois, woik patteins of expeits foi uevelopment of tools foi a uigital
libiaiy, usei piefeiences in seaiching full text uatabases, infoimation seeking in
hypeimeuia uigital libiaiies" (Saiacevic 2uuS, 4). 0seis place the most emphasis on
usability (how easy it is to use) anu usefulness (uoes the libiaiy have the infoimation
neeueu) of a uigital libiaiy. Few stuuies allow useis to contiibute theii own evaluation
ciiteiia.
Selecting ciiteiia foi the evaluation of uigital libiaiies often incluues vaiiations of the
usei anu theii inteiaction with the technology. Theiefoie, a piominent appioach to
evaluation is the usability-centeieu appioach because it focuses on the ease with which a
usei can satisfy theii infoimation neeus with the uigital libiaiy. "0sability-centeieu
appioach: Involves assessment of uiffeient featuies, paiticulaily in iespect to poitals, by
useis. It is a biiuge between systems- anu human-centeieu appioaches" (Saiacevic 2uuS
S). Leainability is also incluueu in many usability stuuies anu is gioupeu with ease of use.
"0sability testing is anothei type of evaluation that focuses on the effects obtaineu when
inuiviuuals apply an infoimation piocessing system to accomplish tasks. 0sability testing
auopts metiics such as time to completion, accuiacy, satisfaction, anu eiiois" (Naichionini
S12). Cuiiently, each ieseaich team selects the usability ciiteiia that they feel woulu be
most impoitant to evaluate; consequently, usability ieseaich measuies a wiue iange of
factois.
Besign of the inteiface plays a majoi iole in usability. "0sability testing often focuses
on assessing the effectiveness, efficiency anuoi satisfaction of the usei expeiience with a
paiticulai inteiface" (Chouuhuiy 1-2). Bespite the quality of content in the uigital libiaiy, if
the usei cannot easily navigate it, they will not use it. Besign elements like "queiy foim,
fielus, instiuctions, iesults uisplays, anu foimats of images anu texts" aie populai
components foi usability stuuies that focus on uesign (Xie 1S49). The uiiect inteiaction
between the usei anu the uigital libiaiy is also stuuieu. The key attiibutes of inteiactivities
aie "iecipiocity, feeuback, immeuiacy, ielevancy, synchionicity, choice, immeision, play,
flow, multiuimensionality, anu contiol" (Xie 1SSu). Reseaich into this aiea laigely
concluues that these piopeities aie inteiuepenuent when evaluating the uigital libiaiy in
teims of usability.
Anothei consiueiation in uigital libiaiy evaluation is whethei it satisfies the usei's
infoimation neeu. Evaluation of the collection focuses on the quality of the collection
examining factois like "completeness, authoiity, accuiacy, ieliable anu ielevant
infoimation" (Xie 1SS9). Regaiuless of the type of libiaiy, its value lies in its ability to
satisfy the usei's infoimation neeu. "If the ietiieval anu usability thiough inteiface uesign
is fantastic but the content is pooi, all of the time spent cieating the uigital libiaiy is
wasteu" (Xie 1SS9). Befining the scope of a uigital libiaiy often helps a usei ueteimine
whethei the collection will meet theii neeus, cieating iealistic expectations of its content.
"The scope of a uigital libiaiy uoes not guaiantee the completeness of the collections;
insteau it is noimally useu to juuge the completeness of the collections of the uigital
libiaiy" (Xie 1S6S). Nany human-centeieu appioaches to evaluation question how well the
uigital libiaiy satisfies the infoimation neeus of a usei oi its community. Foi uigital
libiaiies, satisfying an infoimation neeu maiiies the quality of the collection with the
ability to access it.
0sei satisfaction with a uigital libiaiy is inteiielateu with theii expeiience using
uigital libiaiies anu as a iesult, theii expectations foi use. Xie (2uu7) finus that useis'
"piefeience, expeiience, anu knowleuge stiuctuie co-ueteimine theii evaluation of uigital
libiaiies" (Xie 1S71). "0sei satisfaction is essential foi the success of a uigital libiaiy.
Nultiple channels aie neeueu foi uevelopeis of uigital libiaiies to solicit feeuback fiom
theii taigeteu useis" (Xie 1S6S). Because theie is cuiiently no stanuaiu appioach oi
ciiteiia, the methouology cuiiently employeu vaiies wiuely. Nethous incluue stiuctuieu
inteiviews, focus gioups, obseivations, usage analysis, anu many moie. Chouuhuiy, Bobbs,
anu Loiie iecommenu a multi-attiibute, stateu-piefeience mouel foi evaluating uigital
libiaiy seivices, which suiveys useis anu asks them to state theii piefeiieu seivices oi
featuies fiom a list of choices. "The philosophy unueilying these techniques is that useis
can best assess the appiopiiate tiaueoffs |iegaiuing iesouice allocationj to maximize theii
utility" (Chouuhuiy S). By suiveying useis, libiaiies have auuitional infoimation about
piefeiieu seivices foi uecisions anu can utilize that infoimation to make cost effective
uecisions about seivices that will best satisfy useis.
Theie is some concein that uigital libiaiy evaluation is ielying too heavily on
auapting tiauitional ciiteiia fiom libiaiies anu systems iathei than cieating new ciiteiia
that evaluates aspects unique to uigital libiaiies. Zhang explains "most of the ciiteiia useu
aie meiely boiioweu fiom the uomains of tiauitional libiaiy anu infoimation ietiieval
system. Theie lacks BL-specific evaluation measuies foi examining, foi example, how well
BL infoimation anu collections aie integiateu with each othei, to what extent uiffeient BLs
aie compatible with each othei, how well BLs suppoit socialgioup inteiaction among
heteiogeneous useis utilizing hypeimeuia infoimation, anu whethei theie aie any changes
in useis' uaily woik anu lives that aie associateu with BL applications" (89). Zhang's mouel
incluues ciiteiia that have yet to be incluueu in pievious evaluation stuuies, such as
secuiity, sustainability, anu inteiopeiability, all of which aie consiueieu impoitant
conceins foi uigital mateiials. These ciiteiia waiiant fuithei ieseaich anu inclusion in
futuie evaluation mouels, paiticulaily inteiopeiability, which is only possible with
stanuaiuization.
Tiauitional libiaiies aie similaily oiganizeu to one anothei, which allows foi a
stanuaiuization of ciiteiia. Challenges in ueveloping stanuaiuizeu ciiteiia foi uigital
libiaiies incluue the vaiiety of technology that compiises the uigital libiaiy expeiience anu
the many uiffeient types of useis that the uigital libiaiy may seive. While the vaiiety in
uigital libiaiies cieates uifficulties, it is impoitant to consiuei the neeu foi stanuaiuization
as evaluation techniques aie being uevelopeu. "Nuch moie has to be specifieu anu agieeu
upon befoie uigital libiaiy evaluation can be caiiieu out in a consistent mannei, a mannei
that woulu allow even foi compaiisons" (Saiacevic 2uuu S66). "No agieement exists not
only on ciiteiia, measuies, anu methouologies foi uigital libiaiy evaluation, but even on the
'big' pictuie, the constiuct anu context of evaluation. The evaluation of uigital libiaiies is
still in a foimative stage. Concepts have to be claiifieu fiist. This is the funuamental
challenge foi uigital libiaiy evaluation" (Saiacevic & Covi, S). Noving foiwaiu, uefining
uigital libiaiies anu agieeing upon the fiamewoik oi possible objectives foi evaluation will
help cieate stanuaiuization foi uigital libiaiy evaluation anu potential foi long-teim
evaluation.
Bigital libiaiy evaluation has met with many challenges in effoits to uevelop a
fiamewoik foi consistent evaluation. vaiying peispectives have iesulteu in a lack of a
uefinition foi uigital libiaiies, which in tuin, makes uefining objectives uifficult. Bepenuing
on the ieseaichei's chosen objective, the appioach may be systems-baseu, human-
centeieu, oi a combination of the two, such as usability. Similaily, iueal ciiteiia foi
evaluation woulu incluue tiauitional libiaiy ciiteiia, tiauitional IR ciiteiia, anu tiauitional
human-computei inteiaction ciiteiia as well as ciiteiia unique to uigital libiaiies. Befining
uigital libiaiies, appioaches, ciiteiia, anu methouology piepaie uigital libiaiies foi on-
going, stanuaiuizeu evaluations that enable peei compaiison. Evaluation of uigital
libiaiies ovei time will piesent challenges as technology evolves at a iapiu pace, anu global
accessibility anu inteiopeiability will cieate laigei, moie uiveise usei populations. Bigital
libiaiy evaluation ciiteiia anu methous, once establisheu, will have to iemain flexible anu
evolve to accommouate new technologies anu its useis.


Woiks Citeu

Chouuhuiy, S., Bobbs, B., & Loiie, N. (2uu2). A fiamewoik foi evaluating uigital libiaiy
seivices. B-Lib Nagazine, 8 (78). Retiieveu fiom
http:www.ulib.oigulibjulyu2chouuhuiyu7chouuhuiy.html.

Fuhi, N. et al. (2uu7) Evaluation of uigital libiaiies. !"#$%"&#'("&) +(,%"&) (- .'/'#&)
0'1%&%'$2, 8 (1), 21-S8. Retiieveu fiom
http:static.sciibu.comuocsjSytg1pmxnikf.swf.INITIAL_vIEW=9u
Naichionini, uaiy. (2uuu) "Evaluating Bigital Libiaiies: A Longituuinal anu
Nultifaceteu view." 0'1%&%3 4%$"526 49 (2), Su4-SSS.
Saiacevic, T. (2uuu). Bigital libiaiy evaluation: Towaiu evolution of concepts. 0'1%&%3
4%$"526 78 (2), SSu-S69. Special issue on Evaluation of Bigital Libiaiies. Retiieveu fiom
http:comminfo.iutgeis.euu~tefkoLibiaiyTienus2uuu.puf.
Saiacevic, T. anu Covi, L. (2uuu). Challenges foi Bigital libiaiy Evaluation. 9&:$%
2,1;'##$5 #( #<$ ="",&) >$$#'"/ ?@@@ (- #<$ =;$%'A&" B(A'$#3 -(% !"-(%;&#'(" BA'$"A$C
Retiieveu fiom http:www.is.inf.uni-uue.uecouisesul_ssu4foliensaiacevicuu.puf
Saiacevic, T. (2uuS). Bow weie uigital libiaiies evaluateu. Piesentation at the couise
anu confeience Libiaiies in the Bigital Age (LIBA 2uuS), Su Nay - S }une, Bubiovnik,
Cioatia. Retiieveu fiom http:comminfo.iutgeis.euu~tefkoBL_evaluation_LIBA.puf.
Xie, B. I. (2uu8). 0seis' evaluation of uigital libiaiies (BLs): Theii uses, theii ciiteiia,
anu theii assessment. !"-(%;&#'(" 9%(A$22'"/ D >&"&/$;$"#6 44 (S), 1S46-1S7S.
Zhang, Y. (2u1u). Beveloping a Bolistic Nouel foi Bigital Libiaiy Evaluation. +(,%"&) (-
#<$ =;$%'A&" B(A'$#3 -(% !"-(%;&#'(" BA'$"A$ &"5 4$A<"()(/3, 61(1), 88-11u.