i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

journal homepage: www.intl.elsevierhealth.com/journals/ijmi

An evaluation framework for Health Information Systems: human, organization and technology-t factors (HOT-t)
Maryati Mohd. Yusof a, , Jasna Kuljis b , Anastasia Papazafeiropoulou b , Lampros K. Stergioulas b
a b

Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, 43600 Bangi, Selangor, Malaysia School of Information Systems, Computing and Mathematics, Brunel University, Kingston Lane, Uxbridge, Middlesex UB8 3PH, UK

a r t i c l e
Article history:

i n f o

a b s t r a c t
Background and purpose: The realization of Health Information Systems (HIS) requires rigorous evaluation that addresses technology, human and organization issues. Our review indicates that current evaluation methods evaluate different aspects of HIS and they can be improved upon. A new evaluation framework, human, organization and technology-t (HOT-t) was developed after having conducted a critical appraisal of the ndings of existing HIS evaluation studies. HOT-t builds on previous models of IS evaluationin particular, the IS Success Model and the IT-Organization Fit Model. This paper introduces the new frame-

Received 15 December 2005 Received in revised form 12 August 2007 Accepted 12 August 2007

Keywords: Information Systems Health Information Systems Evaluation Framework Human factors Organizational factors

work for HIS evaluation that incorporates comprehensive dimensions and measures of HIS and provides a technological, human and organizational t. Methods: Literature review on HIS and IS evaluation studies and pilot testing of developed framework. The framework was used to evaluate a Fundus Imaging System (FIS) of a primary care organization in the UK. The case study was conducted through observation, interview and document analysis. Results: The main ndings show that having the right user attitude and skills base together with good leadership, IT-friendly environment and good communication can have positive inuence on the system adoption. Conclusions: Comprehensive, specic evaluation factors, dimensions and measures in the new framework (HOT-t) are applicable in HIS evaluation. The use of such a framework is argued to be useful not only for comprehensive evaluation of the particular FIS system under investigation, but potentially also for any Health Information System in general. 2007 Elsevier Ireland Ltd. All rights reserved.

1.

Introduction

The benets derived from a Health Information Systems (HIS) require rigorous evaluation. It is claimed that organizational and social issues are the main components of such a sys-

tem [1]. The more technology, human and organization t with each other, the greater the potential of HIS. Most existing evaluation studies of HIS focus on technical issues or clinical processes, which do not explain why HIS works well or poorly with a specic user in a specic setting [27].

DOI of original article:10.1016/j.ijmedinf.2007.08.004. Corresponding author. Tel.: +603 8921 6649. E-mail addresses: mmy@ftsm.ukm.my, maryati226@yahoo.com (M.Mohd. Yusof). 1386-5056/$ see front matter 2007 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2007.08.011

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

387

The central purpose of this paper is to show how we derived a framework for HIS evaluation that incorporates comprehensive dimensions and measures of HIS and provides a technological, human and organizational t. A case study demonstrates the application of the proposed framework for describing the critical adoption factors of a particular HIS with a particular user in a particular setting. This proposed framework, human, organization and technology-t (HOT-t) is constructed so that it is capable of being useful in conducting a thorough evaluation study. It is also possible that it will assist researchers and practitioners to unfold and understand the perceived complexity of HIS evaluation. The new framework builds on previous work on the review of HIS evaluation [8,9]. It also makes use of two models of IS evaluation, namely the IS Success Model [10,11] and the IT-Organization Fit Model [12]. This paper is organized as follows. Section 2 discusses the theoretical background of the proposed framework. The aforementioned models are presented to explore their applicability in improving on those used in Health Informatics. This then forms the basis for our rst proposed evaluation framework for HIS presented in Section 3. Section 4 discusses an existing Fundus Imaging System in a primary care organization that is used as a research case study. Section 5 presents the research methodology for applying our framework to this case study. The case study ndings are presented in Section 6. Finally, discussion and conclusions are given in the last section.

2.1.

Human and organizational factors

2.

Theoretical background

The proposed human, organization and technology-t evaluation framework was developed after a critical investigation of the existing ndings of HIS and IS evaluation studies (discussed in the preceding paper in this issue of the journal, [13] and from feedback gained from two conference presentations of earlier stages in the research [8,9]). A review of success determinants of Inpatient Clinical IS indicates that the categories for success in the IS Success Model can be used to assess HIS [14]. The IS Success Model has then been identied as being complementary to another model in fullling the limitations of existing HIS evaluation frameworks, namely the IT-Organization Fit Model. As a result, both models are utilized in constructing HOT-t. The IS Success Model is adopted because of its comprehensive, specic evaluation categories, extensive validation and its applicability to HIS evaluation [10,11]. HOT-t makes use of the IS Success Model in categorizing its evaluation factors, dimensions and measures. In addition, the IT-Organization Fit Model [12] is used to complement the IS Success Model by integrating its featured organizational factors and the concept of t between the human, organizational and technological factors. The two models are discussed in the last two sub-sections of this section prior to HOT-ts development being explained in Section 4. But rst some background to the human and organizational aspects is necessary for later discussions.

The importance of human and organizational factors in the development and implementation of IS has been advocated in the IS literature. Rigorous evaluation of HIS can be undertaken by addressing these factors as well as the t or alignment between them. According to Willcocks [15], the alignment of organization, technology and human is an important starting point in IT implementation as it is one of the strategies that affect IT investment. Barriers to using HIS are also important to consider in HIS evaluation as they explain the failure and success of these systems. Culture and process changes are reported to be the barriers to the wider use of health care systems [16]. Studies cited in Anderson [17] identied a number of barriers to direct physician use of HIS including low level of expertise, lack of acceptance, lack of medical staff sponsorship and alteration of traditional workow patterns. Examples of organizational challenges include hospital culture, such as being risk adverse, reluctance to invest much in IT and resistance to change [18,19]. A study on factors inuencing success and failure of HIS by a group of medical informaticians identied 110 success factors and 27 failure criteria from a wide range of socio-technical issues [20]. These factors are assessed for six types of HIS. Highest success factors for Clinical Information Systems (CIS) are: collaboration and cooperation, setting goals and courses; while for educational system is user acceptance. Highest failure criteria for CIS is response rate; while for administrative system is not understanding the organizational context and not foreseeing the extent to which new HIS affects the organization, its structure and/or work procedures. In short, human and organizational factors are as important as technical issues with regards to system effectiveness [4]. Human, organizational and technical elements should also have a mutual alignment or t in order to ensure successful HIS implementation. It is crucial that HIS t organizational aspects as well as align with work routines, management assumptions, patient care philosophies and users needs as the introduction of a system affects different dimensions of t in complex ways [3]. A number of studies in Health Informatics have included the concept of t in explaining the interdependent relationship between human, organization and technology factors [21,22]. Aarts et al. [21] propose a model which illustrates the stages of information and system changes and their relative specic personal requirements. They argued that changes in both technology and health practices affect each other to a similar extent. Berg [22] makes use of the socio-technical approach in HIS evaluation; work practices are seen as integrated networks of various related elements such as people, tools, organizational processes, machines, and documents. Southon et al. [23] found that the lack of t among main organizational elements contributes to a large number of system failures in public health. The t between technical, organizational, and social factors is analyzed to identify gaps between current health care systems and new system features [24]. Kaplan [3] shows that poor t between system developers goals and clinicians cultural values contributes to user reluctance to use Clinical Decision Support

388

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

Systems (CDSS). Executive support, understanding the business, IT-business relations, and leadership are identied as both enablers and inhibitors of the t of IT and business [25].

2.2.

IS Success Model

In order to structure numerous results from research on IS success categories; a comprehensive taxonomy is introduced [10]. A model is constructed which consists of six success categories or dimensions; they are linked causally and temporally as success is viewed as a dynamic process instead of a static state. The multidimensional relationships among the measures of IS success have been tested extensively in a number of IS studies [26]. Based on these studies, an updated version of this model is presented [11] (see Fig. 1). These measures are included in the six system dimensions: System Quality (the measures of the information processing system itself), Information Quality (the measures of IS output), Service Quality (the measures of technical support or service), Information Use (recipient consumption of the output of IS), User Satisfaction (recipient response to the use of the output of IS) and Net Benets (the overall IS impact). Because of its circularity, the framework might lead to spiral behaviors in both positive and negative directions. For example, effective use of the system will result in higher net benets which lead to more intensive use of the system. In contrast, insufcient system use will yield to lower net benets; thereby acting as a disincentive to system use. In both cases, the IT-Organization Fit Model highlights the important role of the organization in managing the process of changes that take place during system introduction [12]. Effective process management can lead to increased user acceptance and participation in system use, which will in turn help achieve high net benets and subsequently increase system use. In comparison with existing HIS frameworks, DeLone and McLeans IS model illustrates clear, specic dimensions of IS success or effectiveness and the relationships between them. However, it does not include organizational factors that are pertinent to IS evaluation. In addition, Van der Meijden et al. [14] discovered that a number of measures such as user involvement during system development and organizational culture do not match any of the dimensions of the framework. The extension of this framework is recommended by adding the organizational factors, their dimensions and clinical measures related to the healthcare domain.

Fig. 2 The MIT90s (IT-Organization Fit Model) (adapted from Scott Morton, [12]).

2.3.

IT-Organization Fit

Fig. 1 Information System Success Model (Source: DeLone and McLean [11]).

Management in the 1990s (MIT90s) is a well-known ITOrganization Fit Model, which includes both internal and external elements of t [12]. Fig. 2 illustrates the concept of t between the main organizational elements. Internal t is accomplished by a dynamic equilibrium of organizational components including business strategy, organizational structure, management processes, and roles and skills. External t is achieved by formulating organizational strategy based on environmental trends and changes such as market, industry and technology. Within this internal and external t as its enabler, IT is expected to affect the management process, thus impacting on organizational performance and to some degree, its strategy. In order to realize the benets of IT, three prerequisites are required for successful IT transformation. First, organizational vision and the reasons behind it have to be clear to organizational members to get them prepared for organizational changes and hence reduce the challenges in managing transformation. Second, organizational corporate strategy (business and IT), information technology and organizational dimensions have to be aligned with each other. Third, a robust IT infrastructure such as an electronic network and understood standards should be equipped within the organization. These three prerequisites as well as the internal and external t may be used to identify the problems in IT implementation. This model was relatively new and has not been extensively utilized in healthcare [27]. The model was also identied as being capable of identifying the main organizational elements which can affect IS as well as emphasizing the essential alignment or t between them. Moreover, the model is comprehensive as it includes the following factors: technology (IT), human (roles and skills) and organization (strategy, structure and management process). However, these factors can be categorized into more detailed dimensions to provide more specic evaluation dimensions. For instance, IT can be further classied into system quality and information quality, as proposed by DeLone and McLean [10]. Similarly, roles and skills can be associated with use and user satisfaction. Based on the strengths and limitations pointed out in both models, IT-Organization Fit and the IS Success Model complement each other in presenting a comprehensive evaluation framework. Organizational factors, which are lacking in the IS

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

389

Fig. 3 Proposed human-organization-technology t (HOT-t) framework.

Success Model, are featured in the IT-Organization t. Similarly, specic evaluation dimensions and measures which are lacking in the IT-Organization t, are featured in the IS Success Model. Based on the two models explained above, a new evaluation framework is presented in the next section.

3.

Proposed evaluation framework

Satisfaction, Organizational Structure, Organizational Environment and Net Benets. Each of these dimensions is associated with a number of evaluation measures. Examples of evaluation measures according to their corresponding dimension and factor are listed in Table 1. These evaluation dimensions inuenced each other in a temporal and causal way: System Quality, Information Quality and Service Quality singularly and jointly affect both System Use and User Satisfaction. Organizational Structure and Organizational Environment affects System Use. Some of these relationships are two ways: System Use, which relies on user knowledge and training, can inuence the Information Quality, since the users knowledge in using the system can affect reports, images and prescriptions produced by the system. The level of System Use can affect the degree of User Satisfaction and vice versa, for both positive and negative cases. Effective System Use yield to higher User Satisfaction as user is able to explore and make full use of system features and functions; higher User Satisfaction subsequently motivate/lead user to increase System Use. Similarly, the Organizational Environment factors such as government policy and politics can affect Organizational Structure while factors in Organizational Structure will affect the population served in the Organizational Environment. System Use and User Satisfaction are direct antecedents of Net Benets. Net Benets subsequently affect System Use and User Satisfaction. Similarly, Organizational Structure and Environment are direct antecedents of Net Benets. Net Benets subsequently have impact on organizational Structure and Environment. The concept of t is perceived as complex, abstract and subjective. It can be viewed in terms of strategic planning (formulating IS plan according to organizational plan) and strategic alignment (managing IT closely with organizational needs) perspectives [23]. In the HOT-t context, t is concerned with the ability of HIS, human (HIS stakeholders and clinical

Building on previous studies on the evaluation approach [8,9], the proposed evaluation framework was developed after a critical appraisal of the existing ndings of HIS and IS evaluation studies. It makes use of the IS Success Model in categorizing its evaluation factors, dimensions and measures. The IS Success Model is adopted based on its comprehensive, specic evaluation categories, extensive validation and its applicability to HIS evaluation. In addition, the IT-Organization Fit Model is also used to incorporate the concept of t between the evaluation factors: human, organization and technology. The IS Success Model was extended by the addition of the following features which are explained in the following part of this section (see Fig. 3): Organization factors, their dimensions (Structure and Environment) and evaluation measures (listed in Table 1). Fit between technology, human and organization factors. Two-way relationships between these dimensions: Information Quality and System Use, Information Quality and User Satisfaction, Organizational Structure and Environment, Organizational Structure and Net Benets, Organizational Environment and Net Benets. One-way relationship between these dimensions: Structure and System Use. New evaluation measures pertinent to HIS and IS in general. Human, organization and technology are the essential components of IS; the impacts of HIS are assessed in the net benets. These three factors and the impacts of HIS correspond to eight interrelated dimensions of HIS success: System Quality, Information Quality, Service Quality, System Use, User

390

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

Table 1 Examples of the evaluation measures of the proposed framework

Amount/duration:(number of inquiries, amount of connect time, number of functions used, number of records accessed, frequency of access, frequency of report requests, number of reports generated), use by whom? (direct vs. chauffeured use,) actual vs. reported use, nature of use (use for intended purpose, appropriate use, type of information used,) purpose of use, level of use (general vs. specic,) recurring use, report acceptance, percentage used, voluntaries of use, motivation to use, attitude, expectations/belief, knowledge/expertise, acceptance, resistance/reluctance, training

System use

practices) and setting to align with each other. Thus, t can be measured and analyzed from multiple compatibility between human, organization and technology (human-organization, human-technology, organization-technology) using a number of measures dened in the three factors including system exibility, systems ease of use, system usefulness, information relevancy, user attitude, user training, user satisfaction, organizational culture, planning, strategy, management and communication. For example, an effective use of a HIS can be attributed to the t between system exibility and clinical process (system-human). The t between human, organization and technology is illustrated by the bold arrows in Fig. 3. Based on its comprehensive dimensions and outcome measures, the framework could be used to evaluate the performance, effectiveness and impact of HIS or IT in healthcare settings. Effectiveness refers to the accomplishment of specic goals with accuracy and completeness, as well as the correct utilization of appropriate resources [28]. In this research, effectiveness is dened as the ability of a healthcare organization to continuously accomplish goals using optimum resources within a specied time. The three evaluation factors can be evaluated through out the whole system development life cycle namely planning, analysis, design, implementation, operation and maintenance. Meanwhile, Net benets can be anticipated before implementation and apparently evaluated after implementation. As mentioned above, each phase focuses on different issues. This framework can be applied using qualitative, quantitative or a combination of both approaches. Subsequently, any data collection methods from both approaches can be employed while conducting the evaluation. The following subsections explain the evaluation factors, dimensions and measures in detail.

Organization

User satisfaction Human

Satisfaction with specic functions, overall satisfaction, perceived usefulness, enjoyment, software satisfaction, decision making satisfaction

Nature, (type, size) culture, planning, strategy, management, clinical process, autonomy, communication, leadership, top management support, medical sponsorship, champion, mediator, teamwork

Structure

Financing source, government, politics, localization, competition, inter-organizational relationship, population served, external communication

Environment

Clinical practice (Job effects, task performance, productivity, work volume, morale,) efciency, effectiveness (goal achievement, service), decision making quality (analysis, accuracy, time, condence, participation), error reduction, communication, clinical outcomes (patient care, morbidity, mortality,) cost

Net benets

3.1.

Technology

The studies on System Quality are often associated with system performance. System Quality in a healthcare setting measures the inherent features of HIS including system performance and user interface. Examples of system quality measures are ease of use, ease of learning, response time, usefulness, availability, reliability, completeness, system exibility, and security [2933]. Ease of use assesses whether healthcare professionals regard HIS as satisfactory, convenient and pleasant to use. Availability refers to the up time of HIS while exibility is concerned with the ability of HIS to adapt to a healthcare setting and integrate with other systems. Even systems that often work are often not used as anticipated. Thus, it is important to determine whether the system (1) meets the need of the projected users, (2) is convenient and easy to use, (3) ts the work patterns of the professionals for whom it is intended and the overall health system [30]. Measures of Information Quality are concerned with information produced by HIS including patient records, reports, images and prescriptions. Information quality measures can be subjective, as they are derived from the user perspective. Criteria that can be used for HIS quality are information completeness, accuracy, legibility, timeliness, availability, relevancy, consistency and reliability [23,29,30,33,34].

Service quality Information quality System quality Technology

Data accuracy, data currency, Database contents, ease of use, ease of learning, availability, usefulness of system features and functions, exibility, reliability, technical support, security, efciency, resource utilization, response time, turnaround time

Importance, relevance, usefulness, legibility, format, accuracy, conciseness, completeness, reliability, timeliness, data entry methods

Quick responsiveness, assurance, empathy, follow up service, technical support

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

391

Service Quality is concerned with the overall support delivered by the service provider of HIS or technology, regardless of whether the service is delivered by the internal department of healthcare organization or outsourced to external providers. Service quality can be measured through technical support, quick responsiveness, assurance, empathy and follow up service.

[17,23]. The environment of a healthcare organization can be analyzed through its nancing source, government, politics, localization, the type of populations being served, competition, inter-organizational relationship, population served, and communication.

3.4.

Net benets

3.2.

Human

The IS Success Model includes the use of system and user satisfaction while MIT90s incorporates roles and skills as part of human factors elements. HOT-t combines those human factors in both frameworks in addition to other factors of human factors as mentioned earlier. System Use is concerned with the frequency and breadth of HIS inquiries and functions. The use of information output such as reports appears to be one of the most frequent measures to assess the success of IS. The actual use of the system as a measure of IS success refers to voluntary instead of mandatory use. System Use also relates to the person who uses it, their levels of use, training, knowledge, belief, expectation and acceptance or resistance [30,33]. Knowledge is concerned with computer literacy and skills [7,17]. Expectation refers to the anticipation of improved patient care delivery from the use of HIS [7]. Jiang et al. [35] regard resistance as an important factor of system success. As different types of systems are usually related with a particular type of function and user, the reasons for resistance might differ among system types. Resistance can be viewed from one of the following theories: (1) people-oriented, (2) system-oriented and (3) interaction-oriented. People-oriented theory describes resistance to system results from users (groups or individuals) internal factors. Personal characteristics such as age, gender, background, value and belief have been suggested as inuencing individuals attitude towards technology. System oriented theory suggests that resistance results from system design factors or relevant technology including user interface and system characteristics. Interaction theory explains resistance from the interaction between people and system factors; thus, assessment of a system varies across settings and users. Job insecurity and fear are some examples of interaction resistance. User Satisfaction is often used to measure system success. It is subjective in nature as it depends on whose satisfaction is measured. User satisfaction is dened as the overall evaluation of a users experience in using the system and the potential impact of the system. User Satisfaction can be related to users perceived usefulness and attitudes towards HIS which are inuenced by his/her personal characteristics.

A system can benet a single user, a group of users, an organization or an entire industry. Net Benets capture the balance of positive and negative impacts on user, which includes clinicians, managers and IT, staff, system developers, hospitals or the entire healthcare sector. Individual impact is the effect of information on the behavior of the recipient. It is associated with performance as well as changes in user task (clinical practice) such as job performance, change in work activity and improved productivity [7,30]. Thus, individual Net Benets can be assessed using job effects, efciency, effectiveness, decision quality, and error reduction. Organizational impact is the effect of information on organizational performance. In the healthcare context, clinical outcomes can be used as a means of measurement. Examples of these measures include costs reduction, which is due to fewer medication errors and adverse drug effect (ADE); improved efciency in patient care delivery, specically pertaining to tests and drug orders and increased use of generic drug brands and number of consultations and length of waiting lists [7,37]. Clinical outcomes are also measured through two criteria: morbidity (the rate of incidence of a disease) and mortality (death rate). Apart from these quantitative measures, clinical impacts can also be assessed qualitatively using these measures: quality of care, impact on patient care and communication, such as change in communication style and facilitation of information access [38].

4.

The case of Fundus Imaging System (FIS)

3.3.

Organization

The nature of a healthcare institution can be examined from its structure and environment [36]. Organization structure consists of nature including type and size (number of beds), culture, politic, hierarchy, autonomy, planning and control systems, strategy, management and communication. Leadership, top management support and medical staff sponsorship can also be measured from the organization factors

Our research design consists of a case study strategy. The case study serves dual purposes: (1) to evaluate the adoption factors of HIS in the context of the phenomena under study; (2) to validate the proposed HOT-t evaluation framework. The case study is also undertaken to obtain a comprehensive view and understanding of the development process of a HIS described in this section. The case study facilitated the conceptualisation of HIS adoption tending towards success and failure, in association to the factors involved and their relationships. We make use of the case study to answer our research question, as well as to test the applicability of the proposed framework in being a useful evaluation tool. A case study is conducted in the clinical settings of a primary care organization (PCO) and two of its collaborating specialist hospitals, all members of the UK National Health Service (NHS). The GP practice was established in the 1920s and now serves around 6000 patients who are mainly the elderly. The practice is actively engaged in medical and nursing research, training and education. A number of HIS were already in place, including telemedicine and General Practice Information Systems (GPIS).

392

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

This study evaluates the development of a digital Fundus Imaging System (FIS) for diabetic retinopathy. While evaluating FIS, a GPIS was also studied in order to investigate the overall adoption factors of HIS of the specic users in the particular setting. FIS consists of a Fundus camera and digital imaging software that is used to capture the eyes images of patients with diabetes. FIS has functions for image capture, image manipulation, patient data management, image transfer, viewing and temporary storage. For each captured eye image, the system assembles the image, patient data, and timer data into a data le. These data les are transferred to the GPIS by attaching each image to associated patient. The UK government has recommended that all patients with diabetes have their annual retinal screening using digital retinal fundoscopy [39]; the FIS is part of this initiative. FIS is recognized for its importance in preventing blindness among a large population of diabetic patients and in improving the control of diabetic care. Traditionally, retinal fundoscopy for such patients has been carried out in a hospital or by an optometrist. A GP had to refer a patient to a hospital to be invited for annual screening. The recall system in hospital is seen as inefcient in terms of traveling cost and accessibility, particularly to the elderly patient who represents a signicant population of the PCO. The inefciency of traditional screening motivated the partners at the PCO to implement a retinal fundoscopy screening within their practice. The idea to develop the system was initiated by the senior partner, who is a GP himself, after a Fundus camera was purchased. The primary care screening aims to improve the patient pathway by providing better patient educationGPs could educate patients during the screening by showing them their eye images and immediately discuss the clinical implications and subsequently improve their diabetic control. In addition, the patient pathway could be more cost effective through reduced time, effort and expense of travel and double appointments, which allow patients to have both retinal screening and diabetic health check on the same day. The retinal screening of the FIS is done by a GP, Dr. ABC, who is also the sole user of the system. The screening took place in a separate examination room equipped with a digital Fundus camera with a touch screen display, and PCs to transfer patient eye images into patient records which are featured in the GPIS. However, the retinal screening is temporarily discontinued, as the existing camera does not comply with the guideline of the National Screening Committee, which requires more advance equipment. The attempt to implement the FIS was continued by collaborating with a leading specialist hospital. A group of eye specialists who include Consultant Ophthalmologists and a Reading Centre Advisor was liaised with to obtain expert advice in terms of selecting alternative patient pathways, purchasing digital camera and relevant software, and training for using the system and grading the eye images. During this study period, major system development and improvement decisions with regards to the technology procurement, stafng, quality assurance and possible number of patient pathways are still under negotiation.

5.

Methodology

The approach used in this study was that of a subjectivist, case study strategy employing qualitative methods. A formative evaluation was undertaken of the adoption of FIS to identify system problems as they emerged and to improve the system as it was developed [40]. A subjectivist approach was employed in order to gain an extensive understanding of the healthcare context surrounding the FIS through detailed, insightful explanation of the study [34]. Further, qualitative methods were employed to generate a fuller description of the healthcare setting and its cultural issues and to understand why the system functioned well or poorly in a particular setting. The eld study was conducted between April and September 2005 by one of the authors. A number of data collection methods were employed, including interviews, participant observation and document/artifact analysis. A purposeful, snowball sampling method [41] was used in order to gain in-depth information from key informants about the development of the FIS. Participants were identied from the researchers initial contact with individuals known to staff members. After identifying an initial group of participants, a network was built by asking these rst participants to suggest additional participants for interview. In the end 15 participants were found among clinicians, staff and patients who were associated with FIS, both from the primary care organization and from the two specialist hospitals involved in the study. In particular, the list of participants is as follows (Table 2). Our research approach to the evaluation of the FIS consisted of six iterative phases, which included problem identication, the development of an initial evaluation framework, the selection of a research strategy and methods, system evaluation, framework validation, and renement of the evaluation framework (Fig. 4) [34,36]. All phases were completed. Evaluation problems (issues, questions and concerns) were identied through a literature review as well as observations made during an immersion. The immersion was carried out to set the general context of the research, as well as to establish rapport with relevant stakeholders. During this period, in addition to the FIS, the use of a General Practitioner Information Systems was also observed in order to obtain an overall view of the general application and the attitudes of the PCO staff towards Health Informa-

Table 2 List of participants for the case of FIS Participant

User (a GP) Senior partner GP Nurse Physiotherapist IT staff Ophthalmologist Consultant Reading Centre Advisor Patients Total

Total (N)
1 1 2 5 1 1 1 1 2 15

Initial used in data analysis

Dr. ABC Dr. MNO Dr. DEF STU GHI PQR Mr. VWX Dr. JKL

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

393

Fig. 4 Research design (adapted from Friedman and Wyatt [33]; Kaplan [35]).

tion Systems. Initial data collection was gathered during the immersion. An initial evaluation framework was constructed based on the ndings from this rst phase. The research strategy and methods were selected based on the research problem. The original framework in Part I of this paper has been employed initially for the FIS case study. After the conclusion of the study, the framework was eventually validated as well as rened in terms of its evaluation measures based on an appropriate mix of current literature and case study results. The HOT-t framework was used as a guideline in the evaluation of the FIS. During system evaluation, participant observations of daily clinical routines, meetings, discussions and social events took place in different departments, clinics and nursing homes of the primary care organization and specialist department of the collaborating specialist hospitals. A good rapport between the researcher and participants was established at both formal and informal occasions. Participants were aware of the role of the researcher in their clinical settings. During observations and face-to-face interviews, individuals including users, clinicians and IT staff that were involved with the system were queried about their system use and patient pathways. General background information on the primary care organization and its clinical and administrative staff were gathered to obtain a holistic and in-depth view of the clinical setting. This view included aspects of management, facilities, vision, culture, politics, conict, leadership and of the staff in terms of their way of working, communication, relationships and rapport, as well as attitudes towards computer and IT literacy. Meanwhile, patients were queried about their perception about the system. Data were collected on planned occasions as well as spontaneously in a number of iterative cycles. The data were audio- and hand-recorded, transcribed into eld notes, and analyzed. Based on the HOT-t framework, four techniques were used to analyze the results: coding, analytic memos (such as reection notes, displays, and concept maps), and contextual and

narrative analysis [40]. The eld notes were fully transcribed on margin-marked paper. The margins were used to note any reections, themes relevant to HOT-t and statements which were unclear or needed to be conrmed with the participants. The data were coded and categorized under similar themes or concepts of the HOT-t framework and rened through out a series of analyses. These codes corresponded to each factor, dimensions and measures described in the HOT-t framework (see Fig. 3). Further, texts under the same category were compared to identify variations and nuances in meanings. Categories were compared to discover connections between themes. Concept maps were drawn to understand the relationship between the number of concepts involved in the evaluation of FIS. Contextual and narrative analyses were done based on the themes assigned to the codes and these are further covered in the next section of this paper. Two tests were used to establish the quality of this empirical research [42]: 1. Construct validity: establishing correct operational measures for the concepts being studied. This is concerned with exposing and reducing subjectivity, by linking data collection questions and measures to research questions [43]. In this study, the evaluation measures in the interview questions were identied based on the proposed framework and then linked with the research objectives. The results showed that the interview questions addressed each of the research questions to a reasonable degree. 2. Reliability: demonstrating that the operation of the study such as the data collection produced can be repeated with the same results. This was achieved through detailed documentation of procedures and appropriate record keeping [43]. Activities during immersion were recorded in a detailed eldwork log. Potential bias generally acknowledged in qualitative research approach was overcome by conducting a reliability

394

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

test and data triangulation. Triangulation was done through the use of multiple evidences from different sources to conrm the same fact or nding [42]. For example, a certain fact obtained from a GP was conrmed with a different GP as well as the organization documentation and observation. Feedback from participants regarding a particular conclusion in the setting studied is also gathered to reduce bias.

6. Discussion of the ndings from the Fundus Imaging System case

Based on observations that took place during the immersion and pilot study, a number of emerging themes from the framework, namely human, organizational and technological factors and net benets have been identied. These issues are presented in the next sections, which are organized in terms of the three themes.

6.1.

Technology

The general practice where the study took place is somewhat advanced with its use of modern technology and is well equipped with computing and telecommunications devices. Currently, all staff members are using a General Practice Information System which features electronic patient records as well as service and management for GPs. A number of the associate GPs dislike the GPIS. According to Dr. DEF, it is complicated and hard to learn - I hated it! was her literal response. In comparison to the GPIS, the old system that she used in her previous workplace was template based and made typing and data entering much easier. In contrast to the GPIS, the ease of use of the Fundus Imaging System (FIS) can be seen immediately as a user friendly, simple to use interface and straightforward data entry mode. Captured images can be manipulated, stored and attached to the patient record. However, the inefciency of FIS is viewed from its inconvenient storage capacity. The frequent need to delete and transfer les from a small size memory card (attached to the camera) to a hard drive when it gets full was seen as tedious and time consuming. The user, Dr. ABC said it is a pain to keep on deleting the [images on the] memory card when it is full and it gets full quickly because it has small storage capacity. She had to take the card out and upload the patients images to the PC and linked them with the patient records and saved them in a specic folder. The contents of the memory card will be deleted for future use. In addition to system inefciency, slow response time is also seen as a disincentive for using the GPIS. For example, physiotherapist GHI has to recall a number of exercises appropriate to a patients condition in order to suggest which of them should now be followed by the patient. Although she could access their details on the system, she does not use the system because from her point of view, it takes too long for her to print the list of exercises for the patient (2 min). So she requires patients to remember their own lengthy list of exercises. With regards to the system under investigation (FIS), there are some problems related to the less mature technologies involved in its development. For example, the existing camera had to be replaced, and the use of the current FIS was

temporarily halted because the camera did not comply with the National Screening Committee guideline, which was published after the camera was purchased. In terms of Information Quality, two aspects were analysed: relevancy and completeness. The relevancy of the eye images lies in their usefulness in educating patients to take control of their diabetes. In addition to educating patients effectively, it is very important to produce high-quality, accurate images to avoid erroneous diagnosis. However, images taken with the existing camera are not as detailed and as accurate compared to those provided by hospitals. According to the Reading Centre Advisor JKL, in terms of screening standards, the images are not good enough to detect eye abnormalities; however, in practice and for the intended purpose, the images are good enough. Service Quality can be observed and determined from the usage of GPIS. The service providers of the primary care organization comprise of external vendors and a member of internal IT staff. The internal IT specialist is trusted by the senior partner of the practice, Dr. MNO who argued, He is difcult to deal with but he is always right about something. It seems that the service providers give little empathy to the clinicians that they are serving. According to the senior partner, the external service providers do not care about doctors needs at all; they just care about their businesses. It is also perceived that the user also receives too little support from the IT staff when technical assistance is needed. When she requested technical support for a particular camera feature, a member of the IT staff, Mr. PQR responded by saying Theres nothing that can be done about it, adding He started giving me all the jargon.

6.2.

Human

The Fundus Imaging System (for diabetic retinopathy) was developed for a very specic clinical purposeto have a GP user educate patients with diabetes in improving their control of diabetic care. This purpose was viewed as impractical by the collaborating specialists in terms of cost since it is much cheaper to employ technicians to screen the patient via the FIS. This view was not taken into serious consideration until months after FIS was in place, where a large amount of time and effort were already allocated by the GP in planning, developing, training and using the system. According to nurse STU, although initially most of the users of the primary care organization have minimal IT skills, their willingness to use the system was a key contributing factor that put the system in place. As mentioned earlier, some of the GPs were not supportive of the system because, in their view, it was difcult to use. The remaining staff believe that the use of GPIS and other HIS had assisted them in performing their jobs better; tasks were completed faster and communications between staff and doctors were improved. The user of FIS on the other hand has acknowledged her limited IT skills. However, the user was motivated to use the system with the assistance provided by a staff member at the outset of FIS implementation. The barrier to using the system can be seen from the lack of familiarity of the user in using basic le organization functions, such as copying les, and in performing time consuming and tedious tasks such as deleting and transferring les. This process resulted in user

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

395

dissatisfaction because in her view, it is taking a lot of her limited time. Another example of unfamiliarity was seen when a pop up window was displayed and the user said, I hate these messages, I dont understand what they are talking about. Despite her busy schedule, the user attended a series of training organized outside the practice. She needs to be trained as an accredited screener to be able and eligible to examine the images and inform patients of the initial screening results.

6.3.

Organization

The senior partner played a major role in shaping the organizational culture and establishing the use of technology in the organization. He was the leader with a long-term vision and strategy to keep abreast with technology advancement. He also formed a number of collaborations and partnerships with other primary and secondary care organizations, and universities to exploit new technologies in medicine. Organizational readiness has been established in several occasions. For instance, research meeting is held on a regular basis to discuss existing and new technology adoption and clinical concern pertinent to technology, including the FIS. Informal discussions during lunchtime between various staff members, which includes the senior partner, associate GPs and nurses, have build a strong rapport and good communication between them. Everyone is respected and treated equally regardless of their position. Everyone is also encouraged to work as a team in performing their daily tasks. As nurse STU put it, This place is different from other places. The senior partner really made a difference by creating a teamwork spirit. The teamwork spirit among the staff members can be seen in the decision making process. They tend to seek advice from their colleagues who have expertise in an area they themselves do not feel very condent with. For instance, a GP seek advice from the nurses regarding the critical case of a home care resident and a physiotherapist consulted a GP to check on the effect of a drugdrug interaction. In addition to the rather advanced computing facilities, the organization has an environment that is conducive to work in; for example, manual and electronic records are kept in an orderly manner. According to the user, motivation, personality, and encouragement are the catalytic factors that made possible the adoption and use of the system. After the senior partner assigned the role of FIS user to her, he continued to encourage the GP to change her perception about her role as a sole service provider and convincing her that she is capable of using the system. The initiative to use the system was also encouraged by other staff members. Without these factors, Dr. ABC pointed out that . . .. I would not take the role. It was not easy for me to accept that role initially, it was a big change. I was very reluctant because I have a different picture of a GPs role. In her view, a GP only comes in, sees 30 patients, writes the letters, checks the results, and sends them to hospital if necessary. She believed that the senior partners leadership has changed her perception, which made her realized that she can develop professionally and change. Dr. ABC stated that change can be done in two ways: nurtures a person to make him feel that he can do it or it puts him down; for example, you start to criticise your colleague by saying you are

not good enough, you should do more. Usually encouragement will work. Its like the parenting role. She used this example to refer to the initiation of the FIS in this general practice. The uptake of the FIS is also attributable to medical sponsorship in the practice. The user recalled the time when the camera was not being used after it was purchased. The senior partner kept on teasing the user, using his gentle persuasive skill, prompting her to use the system by saying something like Why dont you have a play with it? After a few reluctant initial attempts, the user nally took pictures of a nurses eyes, which was the starting point of diabetic retinopathy screening in this practice. Three types of communication were observed in this clinical setting: patientphysician, physicianphysician and staffphysician. In terms of working style, although each GP and nurse consults their patients in different ways, a commonality in their communication is made obvious: patientphysician relationships in terms of interpersonal communication and eye contact are essential during consultations. The communication between the user and eye specialist is limited by their busy schedule. On several occasions, a joint meeting was postponed by a few months and this had affected the speed of FIS implementation. This lack of communication has resulted in a difference of understanding about project implementation. In addition, the conict of interest between the project leader, the user and the eye specialist team in terms of technology procurement, training cost and the role of user has also delayed the uptake of FIS. After months of meeting and training, the PCO has considered assigning the role of current user, the GP, to an external screener since the time spent by the GP for training, screening and grading would be more expensive than hiring the screener. This situation, however, has been envisaged by one of the Consultant Ophthalmologists earlier and he had informed the user but it was not taken seriously. Meanwhile, although the communication between staff is good, there is a common lack of communication between IT staff and clinicians, which results in conicts. This is an indication of a typical problem of communication gap between technical staff and users. The IT staff also uses jargon that makes the communication more difcult and confusing to the doctors. The senior partner commented that one of the IT staff always leaves us in a mess. He does not tell us exactly what is going on and when the service provider meets the staff, we look like idiots! He also said that, All service providers are rubbish! Focusing on the external environment of this organization, there are some problems with the communication between this primary care and secondary care organization, which include miscommunication and error in reporting. For example, a patient received the wrong medical reportthe pain was reported to be in the left shoulder instead of the right one. In another instance, physiotherapist GHI stated, The communication between hospital and the primary care organization is dreadful in terms of patient medical history. The hospital did not give any referral letter or information that explains the previous medical diagnoses and it is difcult and time consuming to get hold of these documents. I have to guess to identify the causes of the patients pain.

396

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

6.4.

Net benets

In terms of the net benets of the new system, it seems that most of the practitioners have acknowledged certain effects on patient care and clinical practice. According to Dr. ABC, the benets of primary care screening can be assessed through: (1) patients control in their diabetic care, (2) equivalency or better treatment in primary care than the traditional screening in terms of speedy treatment and (3) low error rate and patients satisfaction. It has been recognized that the FIS has the potential to improve the delivery of patient care by providing better patient education. For example, Dr. ABC said to one of the patients that these are your blood vessels, they are so tiny; if theres any problem with them, it will cause major problems with your vision. The cause of the problem is poor diabetic control. She believes that getting patients see their eye pictures may enable them realize that their diabetic control is not good and they are at a risk of losing their vision. According to Dr. ABC, one way to nd out whether the diabetic control is improved by this sort of patient education is by a blood test in the diabetic clinic. The GP quoted an example of an obese patient who took a lot of liberty with his diet and had very bad diabetic control. The GP changed the patients attitude by showing to the patient pictures of his eyes and discussing possible problems that may occur in relation to his diet. As a result, the patient came back, having lost a couple of kilos and told the GP that she effectively made him change the whole way he thought about his diabetes condition. In addition, the senior partner, Dr. MNO said, By her (the GPs) involvement in taking pictures of the patients eyes, she took up a marvellous opportunity to change her patients attitude disorder. That is a fantastic way of inuencing patients behaviour about managing their own health. The partners of this GP practice envisaged that the implementation of FIS will benet patient by saving their time and cost of travelling, thus enable them to be screened earlier. This shorter patient pathway will also reduce possible error rates as data are stored and access directly within the practice. On the other hand, some negative effects have also been recognized. The use of FIS has increased the GPs work volume. She has to spend additional hours in the retinal clinic capturing the eye images, as well as storing and transferring them to the patient notes. In addition, she has to spend a few hours per month for regular training sessions. As a result, there have been certain occasions when, due to accessibility problems in the FIS, she was late for her surgeries.

time and effort required from the busy GP user, as well as poor technical support and user need for assistance. Moreover, the mismatch between clinical processes with technology was illustrated by the GPs busy schedule with time required for training. External t of the practice is achieved by formulating its strategy according to current IT trends and advances. However, the lack of external t in FIS was shown in the non-compliance of current FIS equipment with the National guideline, which caused the system to be discontinued temporarily.

7.

Further discussion and conclusions

6.5.

Fit between human, organization and technology

Based on these three factors, their t with each other has been recognized. The uptake of FIS was contributed by the users strong acceptance and personality to learn using the system (t between human and technology) and the technical support provided by a staff member that acted as a system champion. However, a lack of internal t can also be seen between technology and human: storage inconvenience and

The case of FIS has demonstrated the importance of having the right user attitude and knowledge in order to be able to use the system effectively and efciently. Arguably, choosing the right people to ll the user role is more important than possessing the required skills, as skills can be acquired later. This can be seen from the use of FIS, which was primarily driven by the willingness of the GP to learn and use it after being persuaded by the senior partner and despite her limited computing literacy. Knowledge can be acquired through appropriate training; however, physicians have a very busy schedule. Thus, alternatives such as having replacement physicians can be taken to enable the user to attend this training. The practice is distinguished by being a research practice and having a good leadership. Such organizational culture and leadership has created awareness of technical advancements among the staff members as well as expedited the adoption of HIS, as shown by the uptake of telemedicine, GPIS and FIS. The alignment of the organizational strategy with IT and the existence of up-to-date computing infrastructure have also facilitated the implementation of HIS in the practice. The barriers to system use are contributed by the user perception, ease of use, response time and clinical process. The user was initially reluctant to use the system due to her perception of her role solely as a service provider. GPs have such a heavy daily workload. This explains why response time is crucial in the adoption decision. The same goes with the training issue. It is quite impossible for a GP in the PCO to attend a series of time consuming training sessions while still having to attend her regular surgery sessions. Meanwhile, the communication gap between clinicians and the IT staff is obviously caused by the knowledge gap as well as individual characteristics towards being more sensitive to different stakeholders needs. Communications between all staff at all levels are crucial to ensure that the purposes and benets of an HIS are understood since communication problem can be costly. One way to achieve effective communication is through leadership. In this case study, leadership as well as top management support has proven to be an important starting point and has major inuence in the realization of HIS. In addition to leadership and top management support, medical sponsorship also played a signicant role in changing user perception and encouraging system use. Furthermore, external factors such as government policy can largely affect the viability of HIS. For example, although

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

397

FIS had already been in place and operational, its use had to be halted due to the fact that the camera did not comply with the latest National Guidelines. Thus, it is very important to plan and develop a system and keep it up-to-date in accordance with the latest policy. FIS can potentially improve healthcare delivery by providing better patient education. Other potential benets of FIS can be identied in terms of patient control of their diabetic care and the provision of equivalent or better treatment in primary care than secondary care organizations in terms of speedy treatment and patient satisfaction. A number of challenges have been encountered while applying the HOT-t framework in data analysis. First, there are a number of data that share the same evaluation measures; a careful consideration has to be taken in selecting the most appropriate measures for each data. For example, communication can be classied under organizational structure and environment; problems in using the system can be categorized under of technology factors (ease of use) or human factors (system use). Otherwise, a repetition of the same data in different category can be used where necessary. Second, the classication of data analysis according to the three evaluation factors has affected the ow of the narrative approach of presenting the data and can result in confusion to the reader. In order to validate its usefulness, the proposed framework was tested in a real clinical setting. The system put on the test was a Fundus Imaging System in a NHS primary care organization. The description of the case study and its ndings offer a strong indication of the applicability of the framework to HIS evaluation. A number of critical factors to the adoption of FIS have been identied; factors that had inuenced the adoption negatively include: system usefulness, response time, technical support, empathy of service quality, user perception and user skills. Meanwhile, factors contributing to the positive adoption of FIS include: information relevancy, user attitude, leadership, medical sponsorship, organizational readiness, clinical process and external communication with the inter-organizational system (the eye specialist). The alignment of IT and organizational strategy has led to the initiation of a number of systems, including FIS. The strong willingness of the user to change her perception and clinical practices resulted in the uptake of FIS. However, the adoption of FIS was disrupted with the incompatibility of the system with the National Guidelines, as well as a lack of technical support and limited communication between the technical staff and the collaborating partners of the specialists hospital. We thus conclude that human, organizational and technological factors and the t between them are essential in the realization of FIS. Although our case study focused on a specic setting, the proposed evaluation framework is potentially useful to researchers and practitioners for conducting thorough evaluation studies of other HIS or IT applications in healthcare settings. As proposed here, the framework can and should be applied in a exible way, taking into account different contexts and purposes, stakeholders point of views, phases in system development life cycle, and evaluation methods. The framework is not the solution to any problem; it is a structured debating tool that stakeholders can access in order to know their own system health better.

Summary points What was known before the study: There is a large number HIS evaluation frameworks looking at different aspects of these systems. The existing evaluation methods do not provide explicit evaluation categories. More work on human and organizational issues is called for as most existing evaluation studies of HIS focus on technical issues or clinical processes which do not explain why HIS work well or poorly with a specic user in a specic setting. What the study has added to the knowledge: Previous work on the evaluation of Information Systems is reviewed. An evaluation framework for HIS, which incorporates the concept of t between human, organization and technology (HOT-t), is proposed using a multidisciplinary approach. The application of the proposed evaluation framework is demonstrated in a real-life, practical context where formal evaluation methods have not been or could not have been used. Insights shed from the ndings of the case study that can be used to inform decision making.

Acknowledgements
We gratefully acknowledge the funding received from both the Public Service Department of Malaysia and from the Universiti Kebangsaan Malaysia (National University of Malaysia) that helped sponsor this study.

references

[1] D.F. Sittig, B.L. Hazlehurst, T. Palen, J. Hsu, H. Jimson, M.C. Hornbrook, A clinical information system research agenda for Kaiser Permanente, Permanente J. 6 (3) (2002). [2] B. Kaplan, Evaluating informatics applicationsclinical decision support systems literature review, Int. J. Med. Inf. 64 (1) (2001) 1537. [3] B. Kaplan, Evaluating informatics applications-some alternative approaches: theory, social interactionism, and call for methodological pluralism, Int. J. Med. Inf. 64 (1) (2001) 3956. [4] B. Kaplan, N. Shaw, People, organizational and social issues: evaluation as an exemplar, IMIA 2002 Yearbook (2002) 91102. [5] B. Kaplan, N.T. Shaw, Future directions in evaluation research: people, organizational, and social issues, Methods Inf. Med. 43 (3) (2004) 215231. [6] J. Brender, C. Nohr, P. McNair, Research needs and priorities in health informatics, Int. J. Med. Inf. 5859 (2000) 257289. [7] E. Coiera, Guide to Health Informatics, Hodder Arnold, 2003.

398

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

[8] M.M. Yusof, R.J. Paul, L. Stergioulas, Health Information Systems Evaluation: A focus on Clinical Decision Supports System. Proceedings of the XIX International Congress of the European Federation for Medical Informatics (MIE2005), Geneva, Switzerland, 2005, pp. 855860. [9] M.M. Yusof, R.J. Paul, L.K. Stergioulas, Towards a Framework for Health Information Systems Evaluation. Proceedings of the 39th Hawaii International Conferences on System Sciences, Kauai, Hawaii, USA, 2006. [10] W.H. DeLone, E.R. McLean, Information systems success: the quest for the dependent variable, Inf. Syst. Res. 3 (1) (1992) 6095. [11] W.H. DeLone, E.R. McLean, Measuring e-commerce success: applying the DeLone & McLean Information Systems Success Model, Int. J. Electron. Commerce 9 (1) (2004) 3147. [12] M.S. Scott Morton, The Corporation of the 1990s, Oxford University Press, New York, 1991. [13] M.M. Yusof, A. Papazafeiropoulou, R.J. Paul, L.K. Stergioulas, Investigating evaluation frameworks for health information systems, Int. J. Med. Inf. (2007). [14] M.J. Van der Meijden, H.J. Tange, J. Troost, A. Hasman, Determinants of success of inpatient clinical information systems: a literature review, J. Am. Med. Inform. Assoc. 10 (3) (2003) 235243. [15] L. Willcocks, Managing technology evaluationtechniques and processes, in: R.D. Galliers, B.S.H. Baker (Eds.), Strategic Information Management: Challenges and Strategies in Managing Information Systems, Oxford, 1994, pp. 365381. [16] K.A. Kuhn, D.A. Giuse, J.L. Talmon, The Heidelberg Conference: setting an agenda for the IMIA working Group on Health Information Systems, Int. J. Med. Inf. 69 (23) (2003) 7782. [17] J.G. Anderson, Clearing the way for physicians use of clinical information systems, Commun. ACM 40 (8) (1997) 8390. [18] K. Bottles, Critical choices face healthcare in how to use information technology, Medscape Gen. Med. 1 (1) (1999). [19] N.M. Lorenzi, R.T. Riley, Organizational ISSUES = change, Int. J. Med. Inf. 69 (23) (2003) 197203. [20] J. Brender, E. Ammenwerth, P.N. Nyakanen, J. Talmon, Factors inuencing success and failure of Health Information System a pilot delphi study, Methods Inf. Med. 45 (1) (2006) 125136. [21] J. Aarts, V. Peel, G. Wright, Organizational issues in health informatics: a model approach, Int. J. Med. Inf. 52 (13) (1998) 235242. [22] M. Berg, Patient care information systems and health care work: a sociotechnical approach, Int. J. Med. Inf. 55 (2) (1999) 87101. [23] F.C.G. Southon, C. Sauer, C.G. Grant, Information Technology in Complex Health Services: organizational impediments to successful technology transfer and diffusion, J. Am. Med. Inform. Assoc. 4 (2) (1997) 112124. [24] R. Heeks, D. Mundy, A. Salazar, Why Health Care Information Systems Succeed or Fail, Institute for Development Policy and Management, University of Manchester, 1999. [25] J. Luftman, Assessing business-IT alignment maturity, Commun. AIS 4 (14) (2000) 150. [26] W.H. DeLone, E.R. McLean, Information systems success revisited, in: Proceedings of the 35th Hawaii International

[27]

[28]

[29]

[30]

[31]

[32]

[33]

[34] [35]

[36]

[37]

[38]

[39]

[40]

[41] [42] [43]

Conference on System Sciences (HICCS), Big Island, HI, USA, 2002. F.C.G. Southon, C. Sauer, K. Dampney, Lessons from a failed information systems initiative: issues for complex organizations, Int. J. Med. Inf. 55 (1) (1999) 3346. N.K. Roderer, Outcome measures in clinical information systems evaluation, in: Medinfo. 2004, San Francisco, CA, USA, 2004, pp. 10961100. A.P. Stoop, M. Berg, Integrating quantitative and qualitative methods in patient care information systems evaluation: guidance for the organizational decision maker, Methods Inf. Med. 42 (4) (2004) 458462. J.G. Anderson, C.E. Aydin, Overview: theoretical perspectives and methodologies for the evaluation of health care information systems, in: J.G. Anderson, C.E. Aydin, S.J. Jay (Eds.), Evaluating Health Care Information Systems; Methods and Applications, Sage, Thousand Oaks, CA, 1994, pp. 529. B. Doebbeling, Information Technology and Primary Care at VA: Interdisciplinary Partnership Opportunities for Providers, Managers, and Researchers, Health Service Research & Development Series, 2003. J. Smith, Health Management Information Systems: A Handbook for Decision Makers, Open University Press, Buckingham, 2000. T. Lippeveld, Routine Health Information Systems: The Glue of a Unied Health System, Workshop on Issues and Innovation in Routine Health Information in Developing Countries, Bolger Centre for Leadership Development Potomac, Maryland, 2001. C.P. Friedman, J.C. Wyatt, Evaluation Methods in Medical Informatics, Springer-Verlag, New York, 1997. J.J. Jiang, W.A. Muhanna, G. Klenin, User resistance and strategies for promoting acceptance across system types, Inf. Manage. 37 (1) (2000) 2536. B. Kaplan, Organizational evaluation of medical information resources, in: C.P. Friedman, J.C. Wyatt (Eds.), Evaluation Methods in Medical Informatics, Springer-Verlag, New York, 1997, pp. 255280. A. vant Riet, M. Berg, F. Hidemma, K. Sol, Meeting patients needs with patient information systems: potential benets of qualitative research methods, Int. J. Med. Inf. 64 (1) (2001) 114. N.T. Shaw, CHEATS: a generic information communication technology (ICT) evaluation framework, Comput. Biol. Med. 32 (3) (2002) 209220. The National Screening Committee (NSC), Transfer and Management of Patient Information in Diabetic Retinopathy Screening Programmes Version 2.0, July 2005. Retrieved 2 August, 2005 from: http://www.nscretinopathy.org.uk (2004). B. Kaplan, J. Maxwell, Qualitative research methods for evaluating computer information systems, in: C.E. Aydin, J.G. Anderson, S.J. Jay, S.J. (Ed.), Evaluating Health Care Information Systems; Methods and Applications, Sage, Thousand Oaks, 1994, pp. 4567. M.Q. Patton, Qualitative Research & Evaluation Methods, Sage Publications, Thousand Oaks, 2002. R.K. Yin, Case Study Research: Design and Methods, Sage Publications, Thousand Oaks, 2003. J. Rowley, Using case studies in research, Manage. Res. News 25 (1) (2002) 1627.