Documente Academic
Documente Profesional
Documente Cultură
Software Projects
April 2013
PLEASE TYPE
THE UNIVERSITY OF NEW SOUTH WALES
Thesis/Dissertation Sheet
Title:
Understanding Risks in Off-The-Shelf-Based Custom
Software Projects
Different software development project stakeholders (developers and acquirers) can have different perceptions of risks and how
they should be mitigated. But these differences are not well understood or managed. This general issue occurs in off-the-shelf
(OTS)-based custom software projects, which use and integrate OTS software in developing specialized software for an individual
customer.
This research provides a better understanding of risks to developers and acquirers in OTS-based custom software projects. This
research consisted of five phases:
1. A systematic mapping study to identify OTS-based software acquisition processes.
2. A further mapping study to identify and classify risks related to OTS-based custom development and acquisition.
3. A survey of developers and acquirers in Indonesia to investigate in practice the characteristics of OTS-based risks that
are shared by developers and acquirers.
4. Analysis of the survey data using a risk assessment framework based on stakeholder analysis. This focused on
identifying differences in risk control, impact and responsibility between both stakeholders.
5. A multi-case study (four cases) to provide a deeper and more contextual analysis of these differences.
In general most acquisition risks derive from the same concerns as development risks. However technically related risks are found
less often in acquisition and project management related risks less often in development.
I hereby grant to the University of New South Wales or its agents the right to archive and to make available my thesis or
dissertation in whole or in part in the University libraries in all forms of media, now or here after known, subject to the provisions
of the Copyright Act 1968. I retain all property rights, such as patent rights. I also retain the right to use in future works (such as
articles or books) all or part of this thesis or dissertation.
I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstracts International (this is
applicable to doctoral theses only).
……………………………………..……………… …….………………
Signature Witness Date
The University recognises that there may be exceptional circumstances requiring restrictions on copying or conditions on use.
Requests for restriction for a period of up to 2 years must be made in writing. Requests for a longer period of restriction may be
considered in exceptional circumstances and require the approval of the Dean of Graduate Research.
ii
Declaration of Originality
‘I hereby declare that this submission is my own work and to the best of my knowledge
it contains no materials previously published or written by another person, or substantial
proportions of material which have been accepted for the award of any other degree or
diploma at UNSW or any other educational institution, except where due
acknowledgement is made in the thesis. Any contribution made to the research by
others, with whom I have worked at UNSW or elsewhere, is explicitly acknowledged in
the thesis. I also declare that the intellectual content of this thesis is the product of my
own work, except to the extent that assistance from others in the project's design and
conception or in style, presentation and linguistic expression is acknowledged.’
iii
List of Publications
The following papers were written and published during the candidature. In each case,
the candidate’s contribution was to design and also conduct the study, perform the
analysis, and draft the original manuscript. Co-authoring supervisors and a NICTA
researcher contributed guidance on content and editorial changes on content, linguistic
expression and presentation.
iv
Abstract
In general most acquisition risks derive from the same concerns as development risks.
However technically related risks are found less often in acquisition and project
management related risks less often in development.
vi
Acknowledgements
I would like to thank the following people for their support, without whose help this
thesis would never have been possible. Before all, I want to thank Almighty God for His
love and guidance through this journey.
I owe my deepest gratitude to my supervisors Dr. Liming Zhu, Dr. Mark Staples and
Professor Ross Jeffery for their continuous support, and valuable suggestions and
discussions. I also thank He Zhang for his contributions to ACIS and EASE papers.
Similar thanks goes to my colleagues at the Software Systems Research Group of
NICTA, Paul R., Shukor, Qinghua Lu, Sherry, Liang Zhao, Emam, Yin Kia, Bassem
and Suronapee, for their fellowship and support.
I am particularly grateful to those survey respondents and case study participants who
made this research possible.
vii
Contents
viii
4.4 Survey on Risks of Off-The-Shelf-Based Software Development and
Acquisition .................................................................................................................. 48
4.5 Discussion ........................................................................................................... 53
4.6 Threats to Validity .............................................................................................. 56
4.7 Summary ............................................................................................................. 57
5 Differences in Risk Perceptions .......................................................................... 60
5.1 Introduction ......................................................................................................... 60
5.2 Research Design .................................................................................................. 61
5.3 Collected Data ..................................................................................................... 67
5.4 Results ................................................................................................................. 67
5.5 Discussion ........................................................................................................... 72
5.6 Threats to Validity .............................................................................................. 73
5.7 Summary ............................................................................................................. 74
6 Comparison of Process-related Risk Perceptions between Developers and
Acquirers ......................................................................................................................... 76
6.1 Introduction ......................................................................................................... 76
6.2 Proposed Framework .......................................................................................... 77
6.3 Case Study Design .............................................................................................. 80
6.4 Result .................................................................................................................. 93
6.5 Discussion ......................................................................................................... 109
6.6 Threats to Validity ............................................................................................ 127
6.7 Summary ........................................................................................................... 128
7 Discussion and Conclusion ............................................................................... 132
7.1 Research Contributions ..................................................................................... 133
7.2 Comparison with Related Work ........................................................................ 139
7.3 Implications for Research ................................................................................. 144
7.4 Implications for Practice ................................................................................... 145
7.5 Limitations ........................................................................................................ 145
Bibliography.................................................................................................................. 147
Appendix A: Glossary................................................................................................... 157
Appendix B: Mapped papers for OTS-based software acquisition mapping study ...... 158
Appendix C: Mapped papers for risks of OTS-based and custom software development
and acquisition .............................................................................................................. 165
Appendix D: The framework inputs ............................................................................. 173
ix
Appendix E: Cross-case analysis for data collection 1 and 2, excluded inputs to the
framework and evaluation criteria for risk assessment ................................................. 175
x
List of Figures
Figure 1.1 OTS-based custom software developer and acquirer relationship .................. 1
Figure 1.2 Overview of the research methodology ........................................................... 6
Figure 2.1 Stakeholder power/interest matrix with stakeholder involvement ................ 19
Figure 5.1 A method for analysing differences in off-the-shelf risks between the
developer and acquirer used in the survey ....................................................................... 63
Figure 5.2 Relationship model of respondent, risk, risk control and impact to stakeholder
......................................................................................................................................... 63
Figure 5.3 Risk control/impact matrix, adapted from power/interest matrix [79][33][80]
......................................................................................................................................... 65
Figure 5.4 The developer respondents’ responses mapped to step 1 and 2 of the template
(35 respondents) .............................................................................................................. 68
Figure 5.5 The acquirer respondents’ responses mapped to step 1 and 2 of the template
(34 respondents) .............................................................................................................. 69
Figure 6.1 The proposed framework ............................................................................... 78
Figure 6.2 Risk control/impact matrix with stakeholder involvement ............................ 79
xi
List of Tables
xii
Table 6.2 Mapping between case study questions, research questions and related
propositions ..................................................................................................................... 84
Table 6.3 Case study participants.................................................................................... 86
Table 6.4 Summary of case study participants’ project and participant information .... 90
Table 6.5 Framework outputs (stakeholder involvement and risk priorities) and
agreement of risk responsibilities.................................................................................... 95
Table 6.6. Comparison of evaluation criteria for risk assessment for four participants 105
Table 6.7 Example of cross-case analysis for data collection 1 and 2, excluding
framework inputs and evaluation criteria for risk assessment ...................................... 107
Table 6.8 Classification of key results regarding risk control and impact .................... 113
Table 6.9 Summary of case study evidence and related propositions/literature ........... 125
xiii
1 Introduction
1.1 Background
Software development projects increasingly use off-the-shelf (OTS) products,
integrating them into the systems under development. Software-developing
organisations can avoid building every part of their product software ‘from scratch’ by
reusing technologies available from third parties [1]. Acquiring ‘OTS-based software’,
i.e. software that itself uses OTS software platforms or components, can be less
expensive than acquiring fully custom-developed software. Custom software
development is either performed in-house or contracted software development with
specific requirements for an individual customer [2][3]. OTS-based custom software is
developed using and integrating OTS software in developing specialised software for an
individual customer [4].
acquire
OTS
Developer Acquirer
software
develop
Throughout this thesis, we use software acquisition term definitions from , the IEEE
Standard 1062-1998 Edition: Recommended Practice for Software Acquisition [6] to
define customer/software acquiring organization and software development
1
organisation. An acquirer is defined to be “A person or organisation that acquires or
procures a system or software product (which may be part of a system) from a supplier”
[6:3] and a developer is defined to be “A person or organisation that performs
development activities (including requirements analysis, design, testing through
acceptance) during the software life cycle process” [6:4].
There are differences in risk perceptions between software developers and acquirers,
since they are from different organisations and have different backgrounds, experiences,
needs and expectations [18]. These differences in perception could also occur because
of differences between each party’s goals and structures [19]. Perspective mismatch
could lead to projects not meeting their expected value, quality and schedule [20].
Differences in risk perspectives could increase areas of conflict and obstruct a shared
understanding of risks [16].
Most existing studies and guidance on software acquisition do not explicitly deal with
the acquisition of OTS-based software. For example, the IEEE Standard 1062-1998
Edition: Recommended Practice for Software Acquisition [6] can be applied to software
acquisition process regardless of the size and complexity of the software [6]. However,
this recommended practice is more applicable for fully developed software and must be
tailored to other types of software acquisition [6]. In this thesis we first investigate the
processes of OTS-based software acquisition before investigating the differences in
2
process-related risk perceptions. Process-related risks are defined as “risks that are
related to the overall process, process elements and their interaction” [21:14].
Different stakeholders perceive risk differently [16]; therefore, there are different
perceptions of stakeholder’s risk responsibility. To manage risks effectively, it is
important to identify risks [16], to involve stakeholders [22][8] aiming to take account
of differences in risk perceptions and to identify stakeholders’ risk responsibility [22].
Even though Keil et al. [16] noted the importance of reconciling user and project
manager perceptions of IT project risks and compared the two stakeholders’
perceptions, they did not provide detailed steps for reconciling risks. A recent study has
highlighted the importance of reconciling perspective mismatch in managing processes
of software projects [20]. In this study, Adolph et al. [20] introduced two stages of
reconciling perspectives: converging and validating. However, they [20] offered no
scheme for achieving perspective agreement. Therefore, it is necessary to reconcile
stakeholder perceptions for managing processes and risks in software projects [16][20].
To manage OTS-based custom software projects successfully, the overall goal of this
thesis is to understand how developers and acquirers manage differences in process-
related risk perceptions in OTS-based custom software projects.
3
processes were first identified and better understood. Therefore, the research question
RQ1 is defined as follows.
4
Step 3: Investigating risks in OTS-based custom software projects
Then we investigated risks in OTS-based custom software projects. This began by
identifying, classifying and comparing risks of OTS-based software development and
acquisition using a second mapping study. Then a survey was performed to empirically
investigate the occurrences of the 11 OTS-specific risks in practice. The survey partially
replicated a study by Li et al. [27] on risk management for OTS-based software
development.
As shown in Table 1.1, this research consists of three research methods: systematic
mapping study, survey and case study. There are two different systematic mapping
studies, detailed in Chapter 3 and 4. Further, the survey results were then used in two
studies (Chapter 4 and 5). Two research methods, the survey and the multi-case study
were used to answer RQ2 and RQ3.
Research gap
Research problems
Research importance
Research objectives
Literature
review Step 3: Investigating risks in OTS-based custom
software projects
6
1.5 Research Limitations
To answer RQ2 and RQ3, this research investigated differences in the perception of 11
process-related risks in OTS-based custom software projects relevant to developers and
acquirers (described in Table 4.10 and in Section 4.4.1). Although focusing on these 11
risks could limit the study’s generalisability, we believe that it can increase our
understanding of how to assess risk by analysing differences in risk perceptions between
developers and acquirers. The context of OTS-based software is not narrow - it is now
very common in software development projects1 due to its advantages. These include
rich functionality, dedicated support from OTS software vendors, and less expensive
development and maintenance [28].
This research investigated risk control and impact for developers and acquirers in OTS-
based custom software projects. However, it did not investigate other risk attributes
(such as the probability of loss and the brief description of risks [29]). This research
also only focused on risk assessment, not risk-management planning, risk resolution, or
risk monitoring [30].
In our survey, we asked the survey respondents which stakeholders were affected by
risks and who could influence risks. Even though we did not investigate actual risks from
developers’ and acquirers’ perspectives (without discussing their responses to their
stakeholders), a study of risk perception is an important approach to understand
differences of actual risks between developers and acquirers. The case study investigated
actual risk perceptions.
We identified six OTS-based software acquisition processes from the first mapping
study: decision-making to make or buy software, architectural decision-making, OTS
selection, managing relationships between developers and acquirers, managing
relationships between OTS adoption and acquirers’ organisations, and managing
relationships between OTS vendors and developers. These processes should be
The second mapping study provides a list of 17 risk categories of OTS-based and
custom-based software, not only from developers’ perspective but also from acquirers’
perspective: planning, requirements, design, integration and testing, system lifecycle,
maintenance, project closure, software, OTS products, cost, environment, people,
systems engineering, vendor relationships, project management, contract and legal.
For RQ2 and RQ3 contributions, the survey and case study show that comparing and
discussing risk perceptions about risk control and impact of 11 process-related risks
relevant to developers and acquirers enhance our understanding of risks in OTS-based
custom software projects (described in Table 4.10 and in Section 4.4.1). Our survey
found that: 1) there are similar and different perceptions about risk occurrences in OTS-
based custom software projects between the developer and acquirer perspectives (RQ2
contribution); and 2) there are similarities and differences in respondents’ (developers
and acquirers) perceptions about which stakeholder is affected by and can control risks
(RQ3 contribution). For RQ3 contribution, the case study resulted in 17 detailed
findings related to levels of risk control and impact, and proposed risk priorities (are
detailed in Chapter 6, and summarised in Chapter 7).
This thesis has proposed a framework to analyse differences in risk perceptions and to
assign risk responsibilities between a developer and acquirer (RQ3 contribution). The
framework extends a stakeholder analysis framework [25][12][26][14][13][10] and can
be used to compare, analyse and discuss level of risk control and impact between a
developer and acquirer. The case study results show that all participants and their
stakeholders agreed on proposed risk responsibilities and risk priorities. Furthermore,
our case study finds that a responsibility should also be assigned to the stakeholder who
has the competence and capacity needed to discharge the responsibility [34][35].
8
From a methodological perspective, another contribution of this research is to
demonstrate that stakeholder analysis can be used for each stakeholder to compare,
analyse and discuss their risk perceptions.
Appendix B details mapped papers for software acquisition and OTS-based software
acquisition processes based on type of paper (empirical, experience, conceptual
framework and standard).
Appendix C details mapped papers for risk categories of OTS-based and custom
software development and acquisition based on type of paper (empirical, experience and
conceptual framework).
9
Appendix D presents the framework inputs from the case study participants and their
stakeholder (Chapter 6)
Appendix E presents a cross-case analysis for data collection 1 and 2, excluded inputs
to the framework and evaluation criteria for risk assessment, from the case study
participants (Chapter 6).
10
2 Literature review
This chapter introduces the literature on processes for OTS-based custom software
projects. Then it reviews literature on risk management of OTS-based custom software
projects, stakeholder analysis and research methods in software engineering research.
There are additional OTS-specific processes and roles that must be included in custom
software development [39][40][41]. These processes are: make vs. buy decisions, OTS
selection, OTS product familiarization, learning and understanding OTS products, OTS
integration and vendor interaction [39][40][41]. Li et al. [39] point out that OTS-based
software development can be a part of traditional and custom-development process
lifecycles (e.g. waterfall or evolutionary), combined with some new activities to manage
risks. There are also new roles found in OTS-based software development literature:
OTS team, a team member responsible for interactions with an OTS vendor [40] and
OTS software knowledge keeper [39].
11
The main stages of OTS-based software development are: requirements, design, coding
and integration [40]. For each stage there are OTS-specific and non-OTS activities.
Each stage has activities that are concurrent or iterative not only within an activity but
also across activities.
iii. Coding: Coding deals with writing glue code and interface, and non-OTS coding.
iv. Integration: Integration consists of integration and test, and target system
installation and acceptance test.
12
parameterization, customization, internal revision, and extensive rework. The second
kind is possible modification, which includes five possible levels: none or minimal,
parameterization, customization, programming, and source code. The last kind of
customisation is interface, and which also has five possible values: none,
documentation, API, OO interface, and contract with protocol.
13
Table 2.1 Software acquisition and OTS-based software acquisition process models found in the literature
Model/framework Software acquisition processes Generic software OTS-based OTS specific processes
acquisition software
acquisition
IEEE 1062 Recommended Planning organisational strategy, √ - -
Practice for Software implementing organisation’s
Acquisition [6] process, determining the software
requirements, identifying potential
suppliers, preparing contract
requirements, evaluating proposals
and selecting the supplier, managing
supplier performance, accepting the
software and using the software
ISO/IEC/IEEE 12207 [50] Acquisition preparation, acquisition √ Must be adjusted -
advertisement, supplier selection,
contract agreement, agreement
monitoring, acquirer acceptance,
closure
GARP [51][52] Refer to IEEE 1062 [6] √ - -
MPS.BR Software Acquisition refers to IEEE 1062 [6] and √ - -
[53][54] ISO/IEC/IEEE 12207 [50]
CAP [55] Initialization Component, Execution - √ OTS selection and
Component and Reuse Component evaluation as the basis for
a make-or-buy decision
SAMM [42] Choice phase and implementation - √ Purchase/iterate approach
phase to select OTS products
CSCA [43] Planning, analysing and evaluating, - √ Use make vs. buy
negotiating, managing and reusing decisions implicitly to
select OTS products
14
2.2 Risk Management in OTS-Based Custom Software Projects
This thesis uses the definition of risk proposed by AS/NZS ISO 31000:2009 as “the
effect of uncertainty on objectives” [11:1]. In software projects, we can classify project
risks into generic and project-specific risks [56]. The first are common to all projects
[57][30][58][59] and the latter depend on specific aspects of the project. From the
perspective of technical development processes [17], risks specific to the use of OTS
products may arise from OTS-based custom software project processes (as mentioned in
the previous section) in OTS-based software development [39][40][41] and acquisition
[60]. One of the reasons that make the processes different is a simultaneous process for
OTS selection, system requirements and system design [40][41].
A risk checklist provides a set of risk items that initially helps identify possible sources
of risk and can be used to quickly assess risks associated with a project [65][66]. Boehm
identifies 10 software risk items from project managers and user representatives [30].
Based on Boehm’s [30] risk list, Ropponen and Lyytinen [58] develop six components
15
of software development risk from project manager perspectives. Other risk checklists
are based on taxonomies. For example, the Software Engineering Institute has proposed
a Taxonomy of Software Development Risk [67]. This taxonomy is organised into three
major classes: product engineering, development environment and program constraints.
This risk taxonomy has been implemented in governmental projects [68] and industry
[69] and empirically investigated in Korean projects [70]. The risk taxonomy has also
been used as a baseline to develop a risk taxonomy for software maintenance [61].
Using a literature review, Barki et al. [57] surveyed project managers and identified 23
risk items grouped by five categories. Based on literature, previous experiences, local
circumstances and usage of language of the organisation, Heemstra and Kusters [22]
identified 36 risk items in nine categories to interview each participant of 5 projects
within a Dutch governmental organisation. Based on interviews of 14 Irish software
project managers, Moynihan identified 21 personal constructs relating to risks [71].
Based on three expert group panels from Hong Kong, Finland and the United States,
Schmidt et al. [59] identified 53 risk items grouped by 14 risk categories for the source
of risk. From the above example, there are many risk items that should be considered as
relevant risk items in a software project [30][65]. Furthermore, a risk checklist must
take into account all key stakeholders in a software project [65].
Table 2.2 Risk checklists in the literature extended from Keil et al. [66]
To effectively implement risk management, Schmidt et al. argue that all stakeholders
must cooperate in and be open about risk management [8]. Other studies propose
collaborative risk identification and analysis, by confronting each stakeholder’s
perception of potential risks, discussing risk probabilities and effects, and finally
agreeing upon the most relevant risks and their mitigation strategies [22][23]. The Riskit
method offers a broad and fully documented approach for risk management including
all relevant stakeholders [38][64]. In summary, even though the aforementioned risk
management methods are similar to many other risk management [38][64], these risk
management methods involve all relevant stakeholders in a software project. However,
these risk management methods do not provide detail on how to analyse and compare
differences in risk perceptions between stakeholders.
19
responsibility should also be assigned to the stakeholder who has the competence and
capacity needed to discharge it [34][35].
Even though Keil et al. [16] point out the importance of reconciling user and project
manager perceptions of IT project risks, and comparing stakeholders’ perceptions, their
study does not provide detailed steps for reconciling risks. A recent study has also
highlighted the importance of reconciling perspective mismatch in managing processes
of software project [20]. In this study, Adolph et al. [20] introduce two stages of
reconciling perspectives: converging and validating. However, the authors offer no
detailed explanation for achieving perspective agreement. This thesis addresses this gap
directly.
There are different research classifications in the literature. Based on the purpose of a
piece of research, Robson defines four categories: exploratory (identifying new insights
and generating ideas and hypotheses), descriptive (portraying a phenomenon),
explanatory (investigating an explanation of a phenomenon) and improving (trying to
improve a certain aspect of the studied phenomenon). Another classification is based on
research paradigm. There are three research paradigms that are commonly used in
software engineering research: qualitative, quantitative and mixed [87][88][89][90]. A
qualitative-based study aims to discover trends, patterns and generalization from
explanatory and context information of study participants [88]. A quantitative-based
study aims to find relationships among dependent and independent variables. A mixed
study aims to complement the limitation of each strategy by combining them [87].
This research has used three research methods: systematic mapping study, survey, and
case study. We have performed two different systematic mapping studies, detailed in
Chapter 3 and 4. The survey method was used in two studies (detailed in Chapter 4 and
5). The case study is detailed in Chapter 6. The nature of this research is exploratory and
does not test hypotheses. A qualitative approach was used in this research to address the
complexity of software system development and human behavior as the central role in
21
the software system development [87]. The three research methods were used to
complement each other and strengthen this research.
2.5 Summary
In this chapter, the literature relevant to OTS-based custom software processes and
risks, and stakeholder analysis has been described and analysed. The key points are
highlighted below.
Most of the literature focuses on risks from the software developer organisation’s
perspective, and little attention has been given to software acquirers [7][15].
However, to manage risks effectively, it is important for all software project
stakeholders to understand the risks [16].
An ideal method for risk assessment of OTS-based custom software projects should
identify, analyse and prioritise risks by involving key stakeholders [22][8][23]. It should
also facilitate reconciling differences in risk perception between the developer and
acquirer.
22
Three research methods were selected based on available resources, access to and
control of subjects, and consideration of the research questions [86][85]. A systematic
mapping study was used to answer a type of research question of description and
classification [85]. A survey was used to better understand the normal patterns of
occurrence of the phenomena by asking frequency and distribution questions [85]. A
case study was used to further understand about the phenomena of differences in risk
perception in the context of OTS-based custom software projects. The multiple research
methods complement each other and are appropriate for their purpose.
23
3 Off-The-Shelf-based Software Acquisition Processes
3.1 Introduction
Software development projects increasingly use off-the-shelf (OTS) products,
integrating them into the systems under development. Software-developing
organisations can avoid building every part of their product software ‘from scratch’ by
reusing technologies available from third parties [1]. Acquiring ‘OTS-based software’,
i.e. software that itself uses OTS software platforms or components, can be less
expensive than acquiring fully custom-developed software. However, there is a need to
better understand the OTS-based software acquisition processes.
Most existing studies and guidance on software acquisition do not explicitly deal with
the acquisition of OTS-based software. For example, the IEEE Standard 1062-1998
Edition: Recommended Practice for Software Acquisition can be applied to software
acquisition process regardless of the size and complexity of the software [6]. However,
this recommended practice is more applicable for fully developed software and must be
tailored to other types of software acquisition [6]. Therefore, it motivated us to
investigate the processes of OTS-based software acquisition using a systematic mapping
study.
24
the nature and quantity of evidence in a research area [92]. This chapter presents the
process and results of a mapping study to identify, compare and classify a set of primary
studies of software acquisition and OTS-based software acquisition.
3.2 Method
Our mapping study protocol was created using guidance by Petersen et al [93], adapted
by combining the last two steps of the protocol. There are five steps in the protocol:
definition of research questions, conduct search, screening of papers, keywording using
abstracts, and data extraction and mapping process.
RQ. “What are the similarities and differences between software acquisition and OTS-
based software acquisition from the process perspective?”
The search string defined in this mapping study is based on keywords from the research
question. The keywords ‘software procurement’ and ‘software purchase’ are used as
synonyms for ‘software acquisition’. To extend the search, we also used ‘COTS’, for
commercial-off-the-shelf and ‘OTS’ for off-the-shelf combined with one of the
following strings: ‘acquisition’, ‘procurement’ or ‘purchase’. The mapping study
focuses on processes for OTS-based software acquisition, not processes specific to
acquisition of packaged software. Nevertheless in Appendix B, we show some literature
found for acquisition of packaged software and ERP systems ([24], [71], [78] and [79])
and for OSS acquisition ([41], [53], [93] and [95]). As we focused on OTS-based
25
software acquisition, the term 'CBSE' was not used because it usually refers to OTS-
based software development. We also did not use the keyword “OSS” in the search
strings, because the definition and classification of OTS software also includes OSS
software and not vice versa [5][48].
All of the strings are combined using Boolean ORs and AND to construct the search
string used in this mapping study. The search string is: ‘software acquisition’ OR
‘software procurement’ OR ‘software purchase’ OR ((cots OR ots) AND
(acquisition OR procurement OR purchase)). The search results using the search
string are shown in Table 3.1 describing publication resources, years of publication,
advanced search methods for each of the publication and results. We used Zotero [94], a
bibliography management tool, to manage literature search results.
In order to identify direct evidence from primary studies, we defined in the study
protocol a classification of software acquisition processes based on IEEE 1062
Recommended Practice for Software Acquisition. These processes are: Planning
organisational strategy, implementing organisation’s process, determining the software
requirements, identifying potential suppliers, preparing contract requirements,
evaluating proposals and selecting the supplier, managing supplier performance,
27
accepting the software and using the software. This topic classification was used to map
data extracted from the publications.
During the keywording process, new sub-categories were identified and added into the
topic classification based on screening results that could not be classified into the topic
classification sub-categories but suited the population, intervention and inclusion
criteria. Because the purpose of this mapping study was to identify process similarities
and differences between software acquisition and OTS-based software acquisition, the
mapping study separated publications into two different classes: (generic) software
acquisition and OTS-based software acquisition. After finishing the keywording
process, new topics were added as sub-categories, as shown in Table 3.3. Appendix B
lists the mapping study results with their mapped papers. For the (generic) software
acquisition classification, six new topics were added: decision-making: make vs. buy,
modelling and simulation, software acquisition improvement, process life cycle,
architectural decision-making and managing relationships between developers and
acquirers. Seven new topics were added to the OTS-based software acquisition
classification: decision-making: make vs. buy OTS products vs. use OSS, process life
cycle, architectural decision-making, OTS selection, managing relationships between
OTS adoption and acquirers’ organisations, managing relationships between OTS
vendors and developers, and managing relationships between developers and acquirers.
A process life cycle topic was also added to both of the software acquisition
classifications.
After finishing the keywording and classifying the primary studies based on software
acquisition and OTS-based software acquisition topics, the frequencies of primary
studies were determined, as shown in Table 3.3. Our discussion as follows is based on
this table and on a thorough reading of the identified publications.
To ensure broader coverage [92], we did not perform specific quality assessment on the
selected studies. However, to define quality of the primary studies, we classified the
mapping study results (listed in Table 3.3) as follows: empirical research, experience,
28
conceptual framework and standard. Empirical research papers are combination of
evaluation and validation research papers [95]. Experience papers, in this mapping
study, comprised experience and solution research papers [95]. In addition, we used a
category of conceptual framework instead of opinion and philosophical paper [95]. The
details of this classification can be found in Table 3.4. We added one paper
classification, standard, because we found two standards for software acquisition
[6][50] and one best practice for software acquisition improvement that can be
considered to be a standard [96].
Table 3.4 Research paper classification summarised from the paper classification by
Wieringa and Heerkens [95]
30
3.7 Discussion
This section provides a discussion on the RQ “What are the similarities and differences
between software acquisition and OTS-based software acquisition from process
perspective?”
OTS-based software acquisition is the acquisition of software that itself uses OTS
software platforms or components. We identified OTS-based software acquisition
processes from the literature, and compared them with a process standard for software
acquisition [6]. The differences between these processes concern the acquisition of OTS
products [42][55][43] and also relate to the influence of the use of OTS products on
software development approaches [41][39][40]. Traditionally, software development
starts with system requirements definition, then defines the system architecture, and
continues with implementation. In OTS-based systems development, there is
simultaneous definition and tradeoff among the OTS marketplace, system requirements,
and system architecture and design [41][39][40].
Even though not all the standard software acquisition processes exist among the
software acquisition processes identified from the literature (Table 3.3), both cover the
life cycle [6]. The standard identifies processes for managing supplier performance,
accepting the software and using the software [6], which were not found in the primary
studies. However, the primary studies include implementing the organisation’s process,
determining the software requirements and preparing contract requirements topics,
which are not found in the software acquisition standard. Elgazzar et al. [97] discuss the
planning and contracting phase of OTS-based software acquisition stressing the impact
of OTS on requirements and contract structure.
There are some commonalities between (generic) software acquisition and OTS-based
software acquisition. One common process involves decision-making to make or buy
software, but a particular condition of OTS-based software acquisition is the
consideration of use of third party Commercial Off-the-Shelf (COTS) products [98][98],
Enterprise Resource Planning (ERP) systems [3], and open source software (OSS)
[99][100]. Another commonality concerns making architectural decisions during
software acquisition [101]. These should be suited to the organisation’s needs [101],
corporate governance [102], and system architecture [103]. Another common concern is
31
the nature of the working relationships between developers and acquirers, through
cooperation, integration and establishing familiarity [104][105][106][102].
Two processes were found for generic software acquisition during the literature search.
These processes are not referenced in the software acquisition standards: modelling and
simulation, and software acquisition improvement. However, there was no explicit
mention of these processes within the OTS-based software acquisition literature. These
can be viewed as gaps in the literature.
There are relationships and organisational issues that must be addressed in OTS-based
software acquisition. Two specific issues in OTS-based software acquisition that do not
occur in generic software acquisition concern managing the relationships between
(third-party) OTS vendors and the acquirers’ organisations, and managing relationships
between OTS vendors and developers. In regard to organisational issues, OTS-based
software acquisition must consider several characteristics of the organisation and its
personnel [108]. Finally, a long-lasting and deep partnership relationships between OTS
vendors and developers can provide benefits in commercial negotiations with acquirers
[109].
32
In sum, OTS product selection is a significant process in OTS-based software
acquisition that distinguishes it from generic software acquisition [6]. Existing software
acquisition standards and processes [6] must be adjusted to accommodate the impact of
third-party OTS components in software acquisition.
33
3.8.3 External Validity
External validity refers to the generalisability from this study [90]. Because the
conclusions in the mapping study were only specific for OTS-based software
acquisition processes, external validity threats are not applicable to the mapping study.
3.9 Summary
OTS-based software acquisition is the acquisition of software that itself uses OTS
components or products. We have presented the findings of a systematic mapping study
on OTS-based software acquisition. We have suggested that for OTS-based software
acquisition, changes should be considered for the existing software acquisition process
standard [6].
Both generic and OTS-based software acquisition have the same overall process
lifecycle. The main difference in OTS-based software acquisition is that there is a
relationship between determining the software requirements and OTS selection. Almost
half of publications covering OTS-based software acquisition concern OTS selection.
OTS selection is also related to architectural design making [105]. Together,
architecture and OTS selection criteria are defined and adjusted iteratively to build an
OTS-based system solution [105]. Two additional impacts in OTS-based software
acquisition concern managing relationships between OTS vendors and the acquirers’
organisations, and managing relationships between OTS vendors and developers.
34
4 Risks of OTS-based and Custom Software Development and
Acquisition
This chapter describes two studies of risks of OTS-based custom software projects: a
systematic mapping study and a survey. This chapter addresses RQ2 of this thesis. The
remainder of this chapter is organized as follows. Section 4.1 introduces the general
background. The next section discusses the research design. Then we present systematic
mapping study processes and results, followed by the survey results. We discuss
answers to the research question and threats to validity before concluding.
4.1 Introduction
In a software project, risks affect all stakeholders [8]. The risks of the software project
arise from the beginning of software acquisition process on software acquirer’s side
[8][7]. Most of the literature focuses on the risks from software developer organisation
perspective and little attention has been given to software acquirers [7][15]. However, to
manage risks effectively, it is important for all software project stakeholders to
understand the risks [16]. In addition, there is a need to better understand OTS specific
risks related to technical development processes [17] as a consequence of using OTS
products. In this study we addressed risks to both developers and acquirers of OTS-
based custom software.
This chapter reports the process and results of a mapping study of published papers on
risks in off-the-shelf and custom-based software acquisition and development. The
objective of this mapping study was to identify, classify and compare risks that exist in
OTS and custom-based software projects not only from the software development
viewpoint but also the software acquisition viewpoint. Schmidt et al. argue that software
project risks can best be managed cooperatively between software developers and
acquirers [8]. Consequently, identifying risk components of OTS-based custom
software is expected to give more understanding of risks of OTS-based custom software
projects. In order to investigate risks of OTS-based and custom software development
and acquisition in real world settings, we used the mapping study results to design a
survey of OTS-based custom software project risks that may be relevant to developers
and acquirers. The respondents of the survey were software developers and software
acquirers of Indonesian background.
35
4.2 Research Design
This study used a systematic mapping study to identify, classify and compare risks of
OTS-based and custom software development and acquisition, and then empirically
investigated OTS-specific risks that may be relevant to the developer and acquirer using
a structured online questionnaire of Indonesian developers and acquirers. The
systematic mapping study provides broad evidence of risks of OTS-based custom
software projects. In addition, the mapping study was able to provide a structure to map
primary studies of risks of OTS-based and custom software acquisition, which have
previously been given less attention than risks of OTS-based and custom software
development [7][15]. Our survey provides evidence of the existence of these risks in
practice, and complements a previous study [27] by adding information about the
acquirer perspective. This survey is exploratory and does not test hypotheses.
36
4.2.2 Research Methods
There are two research methods used in this study: a systematic mapping study and a
survey. A systematic mapping study was used for answering RQ2.1. We then compared
and selected the mapping study results to find risks that may be relevant to the
developers and acquirers. A structured online questionnaire was then used to collect
data about the occurrence of these risks. The survey answered RQ2.2 of this chapter. In
the following sections, each method will be detailed.
37
We defined four groups of search strings in this mapping study derived from the
research question keywords, as shown in Table 4.1. ‘COTS’, ‘OTS’, ‘component-based’
and ‘off-the-shelf’ keywords represented the type of systems under study. These
keywords were combined with either ‘system’ or ‘software’ using the AND operator.
These keywords covered broader software-based system development. The keyword
‘development’ summarized activities related to system development life cycle. The
keyword ‘risk’ was used to focus on risk itself and not on factors constituting risk (e.g.
possibility, loss, and hazard). For OTS-based software acquisition risk, the search
strings also included the keywords ‘procurement’ and ‘purchase’ as synonyms of
‘acquisition’. This study is limited to risks in common to COTS, OTS, component-
based software and OSS [27]. Therefore we did not use OSS keyword in search strings
to answer RQ2.1.1 and RQ2.1.2, because the definition and classification of OTS
software also includes OSS software and not vice versa [5][48]. For custom software,
the keywords ‘custom’, ‘contracted’ and ‘bespoke’ were used.
39
different query strings (see Table 4.1) then
only one paper was used.
Description
Publisher
Published year
Paper classification: classified as either empirical research or experience or conceptual
framework [95]
Risks: risks found in the primary study
Risk category: defined categories to group risks in this study
4.3.4 Synthesis
We extracted keywords from abstracts [93] using the categorization scheme. Using the
data extraction results, we formed keywords to create map of the primary studies. We
initially classified risks into seven categories: four categories of OTS-based software
development processes [40] comprising requirement, design, coding, integration and
testing, and 3 non-process categories: OTS product, people, and project management.
Keywords were predefined but were extended with new keywords that were relevant to
the population, intervention and inclusion criteria. During the keywording process, the
category classifications were updated based on screening results that did not match the
proposed category classifications but did match the inclusion criteria.
40
Table 4.4 Search and selection results
41
Table 4.8). Finally, there are 20 risks mapped into 11 risk categories for custom
software acquisition. These results are broadly in line with previous studies [7][15]
pointing out that risks related to software acquisition are less studied than software
development. Appendix C lists the mapping study results with their mapped papers. The
mapping results are described in the following sub-sections.
To ensure broader coverage [92], we did not perform specific quality assessment on the
selected studies. However, to define quality of the primary studies, we classified the
mapping study results (listed in Table 4.5) into a modified paper classification scheme
[95] as follows: empirical research, experience and conceptual framework (as
discussed in Section 3.6).
42
Table 4.5 Mapping study classification based on type of paper
Risk category OTS Development OTS Acquisition Custom Development Custom Acquisition
Em Ex CF Total Em Ex CF Total Em Ex CF Total Em Ex CF Total
Planning 2 2 4 3 3 1 1 2 1 3
Requirement 24 6 5 35 2 2 3 7 5 1 1 7 1 1 2
Design 2 4 6 1 1 1 1
Integration and 19 3 5 27 4 4 1 1
testing
System lifecycle 1 3 4 1 1
Maintenance 7 1 5 13 3 3
Project closure 1 1 1 1
Software 2 2 1 1 1 1
OTS products 7 5 10 22 2 2
Cost 1 1 1 1 2 2
Environment 2 2 4 1 1 2 1 1
People 2 2 4 1 1 1 1 2 1 1
Systems 3 3 1 1
engineering
Vendor 6 2 1 9 2 3 5 6 2 8 3 2 1 6
relationships
Project 2 1 3 4 4
management
Contract 1 1 1 1 1 1
Legal 1 1 1 1 1 1
Total 70 24 39 133 10 2 24 36 21 4 3 28 12 3 5 20
Note: Em: Empirical research paper; Ex: Experience paper; CF: Conceptual framework paper
43
4.3.6 OTS-Based Software Development Risks
Table 4.6 lists 13 risk categories of OTS-based software development. In this mapping
study, there are no risks found related to: project closure, software, project management
and contract.
45
constraints
Vendor 9 Vendor did not provide enough technical support/ training;
relationships Dependency on external parties (vendor lock-in);
Information on the reputation and technical support ability
of vendor were inadequate;
Difficulty in coordinating meetings with key personnel
Legal 1 Dispute and litigation
Total 133
47
4.3.9 Custom Software Acquisition Risks
Table 4.9 lists 11 risk categories of custom software acquisition. In this mapping study,
there are no risks found related to: integration and testing, maintenance, OTS products,
environment, systems engineering, and project management.
48
4.4.1 Survey Design
The questionnaire targeted developers and acquirers of completed OTS-based custom
software projects, using convenience sampling to identify potential respondents.
Convenience sampling is reasonable to use in an exploratory study [113]. The sample
population of the survey was 111 respondents which had a prior academic-industry
relationship with the thesis author. The population consisted of software acquirers
contracting OTS-based software to external software developers and software
organisations developing OTS-based software for their acquirers of different
organisations. Of the 111 respondents invited by e-mail, 69 (62%) completed the survey.
We posted the questionnaire online using Google Docs. The respondents comprised 35
software developers and 34 software acquirers of Indonesian background. Based on their
project descriptions, there were no respondents working on the same project. We
excluded OTS software vendors, who build OTS software, because we wanted to focus on
the software developers and acquirers. They are the two key stakeholders in software
development projects using and integrating OTS software.
The selected risks are derived from studies that can be classified as conceptual
framework [73][116][77][118][119], experience [120] and empirical research
[27][121][122][123][124]. Even though the risks included those derived from
49
conceptual framework and experience papers, we believe these risks may occur
frequently in real world.
As can be seen in Table 4.10, there are no studies justifying the following risks for
acquirers: not adaptable to requirement changes, requirements not negotiable, upgrade
unfeasible, lack of information on vendor and lack of support. Nonetheless, these risks
were selected because the acquirers may be affected by and have influence on these
risks [10][12][13][14].
Table 4.10 OTS-specific project risks relevant to the developer and acquirer
Respondents were asked to answer which risks in Table 4.10 occurred frequently. By
‘frequently’, we meant to investigate the level of presence of risks in real word settings
50
based on 69 completed OTS-based custom software projects. The respondents could
choose one out of 6 alternative answers provided: ‘do not agree at all’, ‘hardly agree’,
‘agree somewhat’, ‘agree mostly’, ‘strongly agree’ and ‘do not know’. To analyse the
results, we coded ‘do not agree at all’ to 1 and ‘strongly agree’ to 5 (see the results in
Table 4.11).
52
4.5 Discussion
In this section, we discuss answers to our research questions arising from the mapping
study and industry survey.
OTS-based software acquisition risks also have the same risks as custom software
development (unrealistic schedules and budgets, changes in business and
organisational environment that affected the project, and underestimated cost
estimation) and custom software acquisition risks (system is not developed according to
the user requirements). These findings indicate that OTS-based software acquisition
risks also have generic risks of software development project, as reported in these
studies [30][57][58][57]. The generic risks can be found in 14 risk categories of OTS-
based software acquisition, except in OTS products category.
53
4.5.3 RQ2.1.3 What risks are related to custom software development?
Custom software development risks include 11 risk categories (listed in Table 4.8).
There is no risk specific only for custom software development. Most risks found in
vendor relationships, project management, contract and legal are associated with either
contracted or outsourced software development. Contracted software development
needs to address these risks that do not exist in in-house software development.
4.5.5 RQ2.1 What risks are related to OTS-based custom software projects?
Taken together, the results of RQ2.1.1 and RQ2.1.2 provide a checklist for identifying
OTS-specific risks, which similar as micro-processes [125][17]. A micro-process
describes the internal details and working of processes [125]. In addition, these results
corroborate the previous studies [72][73] by providing a checklist for identifying OTS-
specific risks for their frameworks of component-based systems. The mapping study
results align with simultaneous processes for OTS selection, system requirements and
system design [41][40] that may increase risks of OTS-based custom software projects.
In four development contexts under study, there are four common risk categories:
planning, requirement, people and vendor relationships. However, poor requirement
definition and analysis is the only generic risk found in all development contexts under
study. OTS-specific risks can be found in all risk categories. The results show that most
risks found in OTS-based software acquisition have the same concern as OTS-based
software development. In both custom software acquisition and development, specific
risks are related to contracted or outsourced software development projects. These
software project risks should be managed cooperatively between the software
acquisition and development [8].
54
These findings provide a better understanding of risk components of OTS-based custom
software projects. The results can serve as a checklist for risk identification [126] to
help to identify more risks [66], and to assess relevancy of risks [126][66].
4.5.6 RQ2.2 Which OTS-specific risks relevant to developers and acquirers occur
frequently?
There are three points of comparison between our survey results and a previous study of
OTS-based software development risk management [27]. Firstly, in this survey, six
infrequently occurring risks are consistent with risks found in the previous study (these
six risks are infrequent) [27]. Secondly, lack of support, which occurred frequently in
this study, is inconsistent with the previous study (this risk is infrequent) [27]. We
added four risks that were not investigated in the previous study [27], as follows:
complicated multi-OTS components arrangement [73][77], insufficient OTS component
documents [118][124], lack of OTS-driven requirements engineering process [73][77]
and reduced control of future evolution of the system [119]. Thus the previous study
[27] did not identify all potential risks; so there is a need to identify and analyse
additional risks [127].
As can be seen in Table 4.11, among the 11 risks under study, more risks occurred more
frequently in software acquisition compared to software development (nine vs. five
risks out of the 11 risks). We observe three kinds of difference between developers and
acquirers. Firstly, there are three risks that occurred frequently in both software
development and acquisition: complicated multi-OTS components arrangement (R4),
lack of OTS-driven requirements engineering process (R6) and reduced control of
future evolution of the system (R11). Secondly some risks occurred frequently in
software acquisition but not in software development: selection effort ill-estimated (R1),
not adaptable to requirement changes (R2), requirements not negotiable (R3),
maintenance planning unfeasible (R7), upgrade unfeasible (R8), and lack of
information on vendor (R9). Thirdly, insufficient OTS component documents (R5) and
lack of vendor technical support and training (R10) are two risks that frequently
occurred in software development but interestingly occurred infrequently in software
acquisition. This probably indicates that developers need more OTS software
documentations, and vendor support and training for integrating OTS software,
compared to acquirers.
55
4.6 Threats to Validity
In this section we discuss construct, internal and external validity for the mapping study
and survey. We followed a previous mapping study [110] to analyse threats to validity
of this mapping study.
56
Providing related information at the beginning of a survey is expected to give
background and context information for the respondents. In addition, this information
may act as an initial filter to ensure that the respondents have needed knowledge and
want to share his/her experience. There were fewer than 10 respondent inquiries before
and after completing the questionnaire. Furthermore, the computer science educational
backgrounds of almost all of the respondents increased confidence of this study for the
respondents in understanding the survey questions.
This survey study has a moderate sample size and was only conducted in Indonesia;
therefore it may not represent risks of OTS-based software development and acquisition
in general. However, there were a total of 69 respondents: 35 represent software
developers and 34 represent software acquirers. The respondents in the sample vary in
organisation sizes and acquirer/customer domains, which may reduce threats to external
validity.
4.7 Summary
In this chapter, we have presented the design and results of two studies of the risks of
OTS-based and custom software development and acquisition: a mapping study and a
survey. The main result of the mapping study is the identification of risks of OTS-based
and custom software development and acquisition. The secondary result of the mapping
study is the classification of risks of OTS-based and custom software development and
acquisition into 17 categories: planning, requirements, design, integration and testing,
system lifecycle, maintenance, project closure, software, OTS products, cost,
environment, people, systems engineering, vendor relationships, project management,
contract and legal. Consistent with previous studies [7][15], our results show that both
software acquisition is less studied than the software development.
Our mapping study identifies generic and specific risks in OTS-based and custom
software acquisition and development. In all four development contexts under study,
57
there are four common risk categories: planning, requirement, people and vendor
relationships. However, poor requirements definition and analysis is the only generic
risk found in all development contexts under study. In both custom software acquisition
and development, specific risks are related to contracted or outsourced software
development projects. The results also show that most risks found in OTS-based
software acquisition have the same concern as OTS-based software development.
Technical-related risks are found less often in acquisition and project management
related risks are found less often in development. Identifying risk components of OTS-
based custom software is a first step in risk management, not only for software
developers but also for software acquirers. Software project risks should be managed
cooperatively between the software acquirers and developers [8].
The results show that more risks occurred more frequently in software acquisition than
the software development. A possible explanation is that acquirers may be more
affected by or have more ability to mitigate these risks. Insufficient OTS component
documents and lack of vendor technical support and training are two risks that
frequently occurred in software development (but not in software acquisition). A
possible explanation for this might be that the limited ability of developers to control
these risks [16][59][128]. These findings provide a better understanding of risks in
OTS-based custom software projects from the developer and acquirer perspectives.
58
Some limitations are worth noting. Although we used five major software-related digital
libraries, relevant primary studies may be present in other digital libraries. The survey
population is Indonesian respondents only, which may limit its external validity.
59
5 Differences in Risk Perceptions
This chapter reports on a study of risk perceptions of developers and acquirers in OTS-
based custom software projects. The study used an online questionnaire-based survey
and compared stakeholders’ perceptions about risk control over and exposure to 11
risks. This chapter addresses RQ3 of this thesis and has two research questions as
follows, within the context of OTS-based custom software projects.
RQ3.1: How are OTS-specific risks perceived by developers and acquirers?
RQ3.2: How should risk responsibilities be defined between developers and acquirers?
This chapter analyses the results of the survey, used in the previous chapter.
The structure of this chapter is as follows. Firstly, we briefly introduce the background to
the study, and describe the overall research design. Then, we describe a method to analyse
the survey data about which stakeholder is affected by and can control each risk. Finally,
we discuss the results and threats to validity, and presents conclusions.
5.1 Introduction
The risks of OTS-based software development and acquisition have been introduced in
the previous chapter. This chapter further analyses the differences in risk perceptions of
developers and acquirer using stakeholder analysis.
Risks associated with a software project affect all stakeholders [9][7][8][10]. Previous
studies have reported that stakeholders tend to perceive the importance of certain risks as
higher than others if they cannot control the risks, and also that different stakeholders tend
to identify risk being caused by other stakeholders [128][16][59]. As different
stakeholders perceive risk differently [16], therefore there are different perceptions of
stakeholder’s responsibility for risks. Stakeholder perceptions vary based on the
individual's or organisation's background and structure, experience, needs and
expectations [18][19]. To manage risks effectively, it is important to involve stakeholders
[22][8], to take account of differences in risk perceptions, and to identify stakeholder
responsibility for risks [22].
This chapter focuses on investigating risks relevant to developers and acquirers, and in
particular differences in the perception of risks by the developers and acquirers. Another
objective of this chapter is to propose risk responsibility based on stakeholder analysis.
60
One approach that accounts for different stakeholder involvement in a project is
stakeholder analysis [25][12][26][14][13][10]. Stakeholder analysis is typically used for
issues such as stakeholder identification, identification of area of interest, stakeholder
contribution and expectation, stakeholder influence, strategy to involve stakeholder and
stakeholder responsibility [25][26].
Table 5.1 11 OTS-specific project risks relevant to the developer and acquirer
respondents
Process Risk
ID Item
Selection and R1 Selection effort ill-estimated
integration R2 Not adaptable to requirement changes
R3 Requirements not negotiable
R4 Complicated multi-OTS components arrangement
R5 Insufficient OTS component documents
R6 Lack of OTS-driven requirements engineering process
Maintenance R7 Maintenance planning unfeasible
R8 Upgrade unfeasible
R9 Lack of information on vendor
R10 Lack of support
R11 Reduced control of future evolution of the system
In our survey we asked about 11 process-related risks that should be relevant not only to
developers but also to acquirers in OTS-based custom software projects. These are
described in Table 5.1. Respondents were asked who was affected by and who could
influence these risks. The respondents could choose one out of four options:
‘developer’, ‘acquirer’, ‘both’ or ‘don’t know’.
62
Figure 5.1 A method for analysing differences in off-the-shelf risks between the
developer and acquirer used in the survey
Figure 5.2 Relationship model of respondent, risk, risk control and impact to stakeholder
63
The method consists of four steps as in the template and one additional step to propose
risk responsibility as follows.
Step 1/‘Risks impacting stakeholders’ counts the number of each kind of
stakeholder affected by each risk.
Step 2/‘Risks stakeholders can control’ counts the number of each kind of
stakeholder who can control each risk.
Step 3/‘Mapping stakeholders from step 1 and 2 into risk control/impact matrix’,
maps the number of each kind of stakeholders from step 1 and 2 into a risk
control/impact matrix (Figure 5.3). This matrix is adapted from a power/interest
matrix [79][33][80] (described in Section 2.3 and depicted in Figure 2.1). As this
survey focused to investigate different risk perceptions between developers and
acquirers, the mapping process was only conducted for two kinds of stakeholders
from step 1 and 2, developer and acquirer. Therefore if a respondent answering
‘both’ to the survey question then they are included as both a developer and an
acquirer. The mapping process is performed separately for developer and acquirer
respondents.
Risks
Stakeholder
R1 R2 ... R11
Step 1 Developer
Risks impacting Acquirer
stakeholders Both
Don’t know
Step 2 Developer
Risks
Acquirer
stakeholder can
Both
control
Don’t know
Step 3 Mapping
Developer
stakeholders
from step 1 and
2 into risk
control/ impact Acquirer
matrix (see
Figure 5.3)
Step 4 Comparing risks using the risk
control/impact matrix
64
High Keep Key player
Risk control satisfied (B) (D)
(Risk stakeholder can control)
Low Minimal Keep
effort (A) informed (C)
Low High
Risk impact
(Risks impacting stakeholder)
Figure 5.3 Risk control/impact matrix, adapted from power/interest matrix [79][33][80]
In order to map the number of each kind of stakeholder (developer and acquirer) from
step 1 and 2 into the risk control/impact matrix, a center point coordinate of the matrix
has first to be defined for each risk under investigation. The horizontal center is half
of the number of respondents answering step 1 (risk impact). The vertical center is
half of the number of respondents answering step 2 (risk control). The following
example demonstrates the mapping process of the developer respondents’ risk
perception about themselves. For example, for the risk of not being adaptable to
requirement changes (R2) in Figure 5.4, the total number of developer respondents
answering step 1 is 34 and step 2 is 35. Hence, the center point coordinate of the risk
control/ impact matrix is (17, 17.5). The number of the developers responding that
they are impacted by (step 1) and can control (step 2) the risk is 13 and 21
respectively. The number of the developers saying that both stakeholders are impacted
by (step 1) and can control (step 2) the risk is 20 and 11 respectively. Therefore, the
total number of the developer respondents that perceive themselves as impacted by
the risk is 33 and that can control risk is 32. So for the risk of not being adaptable to
requirement changes (R2), developers’ perceptions about themselves can be mapped
into high risk control and impact (Key player (D)), because they are greater than their
center points.
Step 4/‘Comparing risks using the risk control/impact matrix’ compares mapped risk
control/impact matrix result between developers and acquirers. This comparison is
performed separately for the developer and acquirer respondents. For example, as can
be seen in Table 5.3, the risk of reduced control of future evolution of the system
(R11) of the developer respondents have developers mapped into Key player (D) and
acquirers mapped into Keep informed (C). These are compared (D:C). The
65
comparison shows that developer respondents perceive themselves to have more
control over the risk than their acquirers.
After the responses of developer and acquirer respondents are mapped to and
compared using the template (as illustrated in Figure 5.1), the completed templates are
then compared to identify stakeholder agreement about risk perceptions. If there are
any differences in risk perceptions between the developer and acquirer respondents,
stakeholder agreement will be decided from partial agreement between both
stakeholder perceptions. The following examples demonstrate partial agreement and
reconciliations of different risk perceptions. The acquirer respondents in Table 5.3
perceive the risk of reduced control of future evolution of the system (R11) for
developers as Keep satisfied (B) and for acquirers as Keep informed (D). To identify
stakeholder agreement about which stakeholder has high risk control, the risk
perceptions of the developer and acquirer respondents to themselves and their
stakeholders are compared (D:C for the developer respondents and B:D for the
acquirer respondents). This comparison reveals that the developer respondents
perceived themselves to have higher risk control compared to their acquirers; while
the acquirer respondents perceived both stakeholders to have high risk control. There
is a partial agreement that the developers are perceived to best control the risk (R11)
because the acquirers perceived themselves and also the developers to have high risk
control. To reconcile this difference, it can be decided that the developers (as
indicated in Part 2 of Table 5.3) can best control the risk of reduced control of future
evolution of the system (R11). Agreement on this might be used as a rationale to
66
assign risk responsibility to the developer. The same procedure is used to reconcile
different risk impact perceptions between developers (both stakeholders are most
impacted by risk R11) and acquirers (the acquirer is most impacted by risk R11). Both
respondents agree that the acquirer has a higher risk impact.
All of the previous steps has been applied and completed for all risks in Table 5.1.
5.4 Results
This section presents results of the survey organized by the comparison method. Figure
5.4 and 5.5 present the questionnaire results mapped to step 1 and 2 of the method
template (see Table 5.2). The results are organized as comparisons of the risk
control/impact matrix between kinds of stakeholder (developers and acquirers)
separately for each survey respondents (the developer and acquirer respondents) in Part
1 of Table 5.3.
67
Step 1: Risks impacting stakeholder
30
25 24
20
20 19 19
18
17 17
16 16
15 15
15 14
13 13
12 12
11
10 10
10 9 9
8 8
7 7
6
5
5 4 4 4 4
3 3
2 2 2 2
1 1 1 1 1
0 0
0
R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11
15 13 13
11 10 10
88 9
10 7 7
5 6 54
3 3 4 4 3 2 343 43
5 2 2
0 0 0 1 1 1 0
0
R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11
Figure 5.4 The developer respondents’ responses mapped to step 1 and 2 of the template
(35 respondents)
68
Step 1: Risks impacting stakeholder
25
20
20
16
15 14 14 14 14 14
13
12 12 12 12
11 11 11
10 10 10 10
10 9
8 8 8 8 888 8
7
6 6 6 6
5 5
5 4
3
2 2 2 2 2 2
1
0
R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11
Figure 5.5 The acquirer respondents’ responses mapped to step 1 and 2 of the template
(34 respondents)
Part 1 of Table 5.3 shows the results of step 4 in the method template (Table 5.2). Part 2
of Table 5.3 shows stakeholder agreement on risk control and impact, and risk
responsibility following step 5 in the developed method. Table 5.4 summarizes patterns of
the comparisons from Part 1 of Table 5.3.
It can be seen from Table 5.3 and 5.4, almost all the developer respondents perceive
themselves to have higher risk control compared to their acquirers. There are 3 kinds of
differences. The first is D:D, which in Table 5.3 is the risk of requirements not
69
negotiable (R3), where the developer respondents perceive themselves and their
acquirers to have the same high risk control and impact. For the second kind, developers
perceive themselves to have higher risk control compared to their acquirers, but
perceive themselves to have equal risk impact as their acquirers (D:C in Table 5.3 and
5.4). Risks in this group are selection effort ill-estimated (R1), not adaptable to
requirement changes (R2), complicated multi-OTS components arrangement (R4), and
maintenance planning unfeasible (R7) and reduced control of future evolution of the
system (R11). For the last kind, the developer respondents perceive themselves to have
higher risk control and impact compared to their acquirers (D:A in Table 5.3 and 5.4),
for the risks: insufficient OTS component documents (R5), lack of OTS-driven
requirements engineering process (R6), upgrade unfeasible (R8), lack of information on
vendor (R9) and lack of support (R10).
Table 5.4 Summary of comparison of risks mapped into the risk control/impact matrix
between the developer and acquirer respondents (summarized from Table 5.3)
Patterns of comparison of risk mapped into the
risk control/impact matrix (developer:acquirer)
from respondent perspectives
D:D D:C D:A B:C B:D
Respondent Developer 1 5 5
perspective Acquirer 4 5 1 1
70
For acquirers, there are 4 kinds of differences. For the first kind, acquirer respondents
perceive the following risks to be under high control and have high impact for
themselves and their developers (D:D): selection effort ill-estimated (R1), requirements
not negotiable (R3), complicated multi-OTS components arrangement (R4) and lack of
OTS-driven requirements (R6). For the second kind, acquirers perceive their developers
to have higher risk control, and perceive themselves and their developer to have higher
risk impact (D:C) for the following risks: not adaptable to requirement changes (R2),
insufficient OTS component documents (R5), maintenance planning unfeasible (R7),
upgrade unfeasible (R8) and lack of information on vendor (R9). For the third kind,
acquirers perceive themselves to have higher risk impact but lower risk control
compared to their developers (B:C), for the risk of lack of support (R11). Finally,
acquirers perceive themselves to have higher risk impact than, but equal risk control to
their developer (B:D) for the risk of reduced control of future evolution of the system
(R11).
Part 2 of Table 5.3 shows agreements of responses of stakeholders on risk control and
impact, including reconciliation results of different risk perceptions between the
developer and acquirer respondents on the risks of R1, R4, R5, R6, R8, R9, and R11.
With regard to risks R1, R4, R6 and R11, the developer respondents perceived
themselves to have higher risk control than their acquirers, and the acquirer respondents
perceived both stakeholders to have high risk control and impact (for R1, R4 and R6) or
to have high risk control (for R11). The differences between the stakeholders can be
reconciled by agreeing that the developer has a high level of risk control. For the risks
R5, R6, R8 and R9, the developer respondents perceived only themselves as most
impacted by these risks (D:A in developer respondents row in Part 1 of Table 5.3), and
the acquirer respondents perceived both stakeholders as most impacted by these risks
(developer and acquirer of the mapped risk control/impact matrix is either C or D in
Part 1 Table 5.3). To reconcile these differences, it could be decided that the developers
are highly impacted by these risks (presented in Part 2 of Table 5.3). For risk R11, the
responses of both stakeholders agree that the acquirer is most impacted by the risk
(reduced control of future evolution of the system). This agreement is decided by
reconciling different risk perceptions between the developer (both stakeholders are most
impacted by risk R11) and acquirer (the acquirer is most impacted by risk R11). For the
risk of lack of support (R10), there was disagreement about who is most impacted by
71
the risk (developers perceive themselves to have higher risk control and impact
compared to their acquirer (D:A), while acquirers perceive themselves to have higher
risk impact compared to their developers (B:C).
5.5 Discussion
5.5.1 RQ3.1: How are OTS-specific risks perceived by the developer and
acquirer?
Although a previous study investigated differences in perceptions of IT project risks
between users and project managers [16], these risks were not specific to OTS-based
custom software projects. Our study focused on OTS-specific project risks relevant to
the developer and acquirer, as listed in Table 5.1. In this section, we compared risk
perceptions in a risk control/impact matrix. This comparison focused on perceptions of
risk control and impact by developers and acquirers.
As indicated in Part 2 of Table 5.3, almost all stakeholders agree that the developer can
best control risks. There is only one risk, R3 (requirement not negotiable), where both
stakeholders each claim to best control this risk.
The developer respondents have two different kinds of perceptions about which
stakeholder is most impacted by risks. The first, developers perceive that both
stakeholders are most impacted by risks (as shown in Part 1 of Table 5.3, for the risks
R1, R2, R3, R4, R7 and R11). The second kind, developers perceive themselves to be
most impacted by risks (as shown in Part 1 of Table 5.3, for the risks R5, R6, R8, R9
and R10).
5.5.2 RQ3.2: How to define risk responsibility between the developers and
acquirers?
As can be seen in Table 5.3, for only 3 out of 11 risks investigated, (R2, R3, R7) do
both the developers and acquirers have the same perceptions. For the other risks, it is
interesting to analyse the differences.
Understanding these differences in perceptions about which stakeholder has high risk
control is helpful to propose risk responsibility. High influence and power is needed to
control key decisions and task implementation [31][32][33] to mitigate risks. This
study provided a structured method to compare risk control and impact to identify
stakeholder agreement on and reconcile differences in risk perceptions between
developer and acquirer respondents. Ultimately, the outputs of the method are proposals
for the assignment of risk responsibilities, based on stakeholder agreement on which
stakeholder has high risk control. This is consistent with responsibility models
[81][78][34][35], which propose responsibility belongs to stakeholders who have the
competence and capacity needed to discharge the responsibility [34][35]. Knowing the
responsibility could subsequently lead to the identification of roles [81][78] [34][35] to
mitigate risks.
As shown in Part 2 of Table 5.3, for all risks except requirements not negotiable (R3),
the developer would be considered to be responsible for the risks. The developer should
inform, consult and involve the acquirer because the acquirer is also impacted by the
risks [33]. For the risk of requirements not negotiable (R3), both stakeholders should
collaborate to discharge the responsibility [34] of risk mitigation.
73
questionnaire questions from a stakeholder definition, which is anyone who are affected
by or can influence the system under development [12][14][13][10]. The use of
stakeholder definition as the questionnaire questions to ensure adequate definitions and
measures of variables [90].
5.7 Summary
This study attempted to analyse differences in shared risks [8] of OTS-based custom
software projects by comparing perceptions of risk control and impact of two main
stakeholders, the developers and acquirers. We performed an online questionnaire-based
survey about OTS-based custom software project risks on Indonesian software
developers and acquirers. In the survey, we asked who was affected by and who could
influence 11 risks of OTS-based custom software projects relevant to both developers
and acquirers. To analyse the survey results, we developed and applied a method for
analysing differences in perceptions of OTS-specific risks based on stakeholder analysis
[25][12] [26][14] [13][10].
The findings show some differences in risk perception. Both stakeholders can control,
and are most impacted by, risks about requirements negotiation. Developer respondents
perceived themselves to best control risks, but perceived either themselves or both
stakeholders to be most impacted by risks. Acquirer respondents agreed that their
developers can best control risks, but perceived both stakeholders as most impacted by
risks, except for risks of R10 and R11. For the risk of lack of vendor technical support
and training (R10), there was disagreement about who is most impacted by the risk
(usually each stakeholder reported themselves). With regard to the risk of reduced
control of future evolution of the system (R11), both stakeholders agreed that the
acquirer is most impacted.
The comparison method was developed to analyse the survey results. We used the
method to propose a default position about which stakeholder should be responsible for
risks based on stakeholders’ agreement about which stakeholders have high control and
power over each risk [31][32][33]. The results show that for all risks (except
requirements not negotiable) the responsibility can be proposed to be the developers’.
With regard to requirements not negotiable, we proposed both stakeholders had
responsibility of risk.
74
Even though we did not investigate the developer and acquirer’s actual perspectives on
risks, a study of risk perception is an important approach to understand differences of
actual risks between the developers and acquirers. The comparison method analysed
respondents’ responses about their risk control and impact, without discussing their
responses to their stakeholders. Overall, the comparison method in this study added
substantially to our understanding to provide a method-based consideration to analyse
different risk perceptions [16].
75
6 Comparison of Process-related Risk Perceptions between
Developers and Acquirers
The first section introduces this chapter. The next section describes a proposed
framework, which was modified from a method in the previous chapter, to collect,
compare and analyse the case study data. Then, the case study design is detailed.
Section 6.4 presents results and findings of this study. Section 6.5 discusses answers to
the research questions. Finally we assess threats to validity and conclude.
6.1 Introduction
To manage risks effectively, it is important to identify the risks involved [16], to
involve stakeholders [22][8] aiming to take account of differences in risk perceptions
and to identify stakeholder responsibility for risks [22]. Even though Keil et al. [16]
pointed out the importance of reconciling user and project manager perceptions of IT
project risks and compared two stakeholders’ perceptions, their study did not provide
detailed steps for reconciling risks. A recent study has highlighted the importance of
reconciling perspective mismatch in managing software processes [20]. In their study,
Adolph et al. [20] introduced two stages for reconciling perspectives: converging and
validating. In this thesis, we analyse differences in the perception of process-related
risks between the developers and acquirers in OTS-based custom software projects
using a multi-case study.
76
not adaptable to requirement changes, requirements not negotiable, complicated multi-
OTS components arrangement, insufficient OTS component documents and lack of OTS-
driven requirements engineering process. Five are maintenance-related risks:
maintenance planning unfeasible, upgrade unfeasible, lack of information on vendor,
lack of support and reduced control of future evolution of the system. Section 4.4.1
provides a more detailed explanation of these 11 risks. Although focusing on the 11
risks could limit the study’s generalisability, we believe that it can help increase our
understanding to better assess risks through detailed analysing of differences in risk
perceptions between developers and acquirers.
Based on the above research questions and access to resources [85][86], which in this
study were case study participants, a case study was selected as the research method.
Using a case study was expected to provide a deeper understanding of the phenomena
under study [86][85][129] by involving stakeholders in risk management and analysing
differences in risk perceptions occurring in real contexts.
Table 6.1 Differences between the case study framework and the method used in
Chapter 5
This framework analyses differences in perceived risks that should be relevant to both
developers and acquirers of OTS-based custom software. The risks are described in
Section 4.4.1 and in Table 4.10. The framework extends the stakeholder analysis
approach [10][12][13][25][26], and is illustrated in Figure 6.1. The framework
compares level of stakeholder perceptions about risk control and impact (low/high),
which are both framework inputs. Framework outputs are stakeholder involvement and
risk priority (for each stakeholder). Responsibility for a risk is proposed to belong to a
stakeholder who has higher risk control. These outputs and risk responsibilities are then
raised with the developer and acquirer for their agreement. The framework is not a risk
identification method; stakeholders must provide a list of risks to be analysed using the
framework.
78
2. The framework inputs are then mapped into a risk impact/control matrix (Figure
6.2), as in the previous chapter [79][33][80] (see Figure 5.3). The mapping process
output is a value for each risk: inform (A), involve (B), consult (C) and collaborate
(D). Each value indicates the kind of stakeholder involvement. The matrix uses an
ordinal scale. Instead of using risk-exposure quantity [30] or multiplication on
ordinal scale [38], the framework prioritises risk using available information [38] by
comparing the values of two quadrants. Cells above or right have higher risk
priority. Quadrant A has low risk priority. Quadrant B and C have medium risk
priority, and quadrant D has high risk priority. However, risks in quadrant B and C
cannot be compared [69][130].
79
proposed risk responsibility by considering previous results. The discussion process
should skip areas of agreement, which are stakeholder-owned perspectives (inputs
and outputs), and focus on points of difference, which are proposed risk
responsibilities.
Although the framework was designed to be used by a developer and acquirer, to avoid
the effects of learning to use the framework [131], case study participants were only
asked to complete the framework inputs and to provide agreement on the framework
outputs and proposed risk responsibilities (these are detailed in following sections).
To increase validity, a new proposed framework should be evaluated [131] and then
improved [129] using retrospective data [132]. The framework was initially evaluated
using a set of evaluation criteria for risk assessment [133][134] (the results in Section
6.4.2 and discussed in Section 6.5.3), and using the results and findings of this case
study. Finally, the framework was improved [129] by comparing [135] existing risk
management practices with the framework, and making use suggestions from all
participants.
The case study followed existing guidelines [86][131][132][129]. The structure of the
case study is outlined below.
1. Case study design
The case study design comprises:
a. Objectives of the case study
b. Case study questions to guide data collection
c. Theoretical prepositions as term of reference to conduct and review the
study
d. The case and unit of analysis selection
e. Methods to collect data
80
2. Preparation for data collection:
Procedures and protocols for data collection must be defined.
3. Collecting evidence
4. Analysis of collected data
5. Reporting
The following section describes the case study according to this structure.
81
Q1.2.5 How was your level of risk control and risk impact for each of these risks?
(low/high)
Q1.3 (Part 3 of the questionnaire)
The questions collected participant information: participant position in the project (unit
of analysis), participant position in their organisation, experience in software
development projects (in years), experience of using OTS in projects (in years), number
of OTS-based software development projects involved with, educational background
(degree), software engineering/computer science/information system/information
technology/telematics related education, business ownership, industry domain (IT
industry, telecommunication service provider, software vendor/producer, IT
consultant/software house, software-related company, other).
2. Second stage
There were two main questions for the interview: an initial evaluation of the framework,
and analysing differences in risk perceptions, as follows.
Q2.1. Evaluation of the framework
Participants would be asked to initially evaluate the proposed framework using
previously proposed evaluation criteria for risk assessment [133][134]. Four criteria
were used in this study:
Adaptability:
Q2.1.1 How difficult is it to use the proposed framework in your different
projects? (Portability) (easy/medium/difficult)
Q2.1.2 How difficult is it to integrate the proposed framework with your
existing risk management? (Modifiability) (easy/medium/difficult)
Feasibility (of the input data):
Q2.1.3 How difficult is it to get sufficient input data to use the proposed
framework? (Availability) (easy/medium/difficult)
Q2.1.4 How varying are groups of risks as the input data for the proposed
framework? (Scope of the input data) (easy/medium/difficult)
Completeness:
Q2.1.5 Does the proposed framework provide enough level of detail for risks?
(Scope of completeness of the proposed framework) (easy/medium/difficult)
Q2.1.6 Does the proposed framework cover comprehensive risks needed to be
addressed? (Element (of risk))
82
Validity:
Q2.1.7 How do the framework outputs represent the real phenomenon?
(Relevance)
Q2.1.8 Could you implement the framework outputs? (Practicability)
The above questions used related attributes and metrics to measure the proposed
framework [133][134]. The metric was extended into 3-points scale (easy-medium-
difficult) to investigate the degree of existence of an attribute [133].
83
strategies should be used to analyse data [86][85]. This case study used five
propositions (P):
P1 Risks can be managed by involving all stakeholders [8].
P2 Understanding differences of perceptions of OTS-specific risks between developers
and acquirers can help to avoid conflict and thus help to ensure successful software
projects [16].
P3 Perspective mismatch occurs because of differences between software developers
and acquirers as they have different backgrounds, experiences, needs, expectations
[18], and goals and structures [19].
P4 Risk responsibilities are assigned to a stakeholder who has the competence and
capacity needed to discharge the responsibility [34][35][78].
P5 Consensus building can be facilitated by comparing and discussing perspectives
between stakeholders [22][23][24].
Table 6.2 shows mapping between questions of data collection 1 and 2, the case study
questions and the related propositions.
Table 6.2 Mapping between case study questions, research questions and related
propositions
The selection of cases was derived from research questions and access to resources
[86][85]. This case study used purposive sampling from the survey respondents (see
Chapters 4 and 5). One of the benefits having sampling from our survey’s respondents
was their familiarity with the 11 process-related risks. The choice of study participants
was based on a prior academic-industry relationship with the thesis author.
Four case study participants (shown in Table 6.3) were also respondents in our previous
survey. There were 69 total survey respondents. Forty-two potential survey respondents,
who were selected based on their strong relationship with the thesis author, were invited
to participate in this study and eight invited respondents agreed to participate. For the
purpose of data collection in this study, described in Section 6.3.6, the participants were
asked to involve their stakeholder. Of the eight respondents, only four could involve
their stakeholder to provide inputs for the framework. Therefore, four participants were
finally selected for this study. The case study participants and their stakeholders varied
in terms of their organisation sizes and background. Based on participatory
arrangements, the cases are referred to as 1Dev, 2AcqBank, 3AcqTelco and 4AcqGovt.
The participants were the main contact persons for the case study. The participants were
involved in all stages of the data collections. Each of their stakeholders came from
different organisations, and was only involved in providing inputs to the framework (see
question Q1.2.5) and participating in the agreement process for framework outputs after
the interview. Three stakeholders (not the one for 3AcqTelco) also gave detailed risk
perceptions along with the inputs to the framework. In the case of 1Dev and 2AcqBank,
85
detailed risk perceptions were collected using different interviews. The stakeholder of
the 2AcqBank participant was also involved in the second data collection interview,
where the agreement of the proposed risk responsibilities occurred. The other three
stakeholders agreed to the proposed risk responsibilities after the interview.
The project stakeholders included one acquirer and three developers, and were known
from the survey to have met the requirements to participate in this research.
86
1. Case 1: 1Dev
The participant of case 1Dev is a domestic software developer company. This software
developer organisation has experience in customer relationship management, contract
management, dashboard application, workflow management system, interactive
multimedia and system integration. Their clients include government, banks, publishers,
postal service, university and telecommunication companies. The participant’s
stakeholder organisation is an acquirer. It is an information technology division of an
Indonesian telecommunication operator. It functions to support products and services
for internal usage and for external clients. The division was established in 1977 to
support and automate billing processes.
In this case, the 11 OTS-specific risks were investigated on a single sign on application.
The application was built for one a telecommunication operator in Indonesia (acquirer).
The application was used to increase interoperability and extensibility among existing
customer care systems by integrating them into one interface and single sign on
application. This application was a simplification and unification of pre-sales, sales and
after-sales functions in customer relationship management. Therefore, this application is
a critical application serving all contact points of the acquirer's service centers for all
customer segments and all types of services by minimising delay and transfer of
services. The application was built using a commercial PHP server, Oracle database,
JavaScript and commercial active directory. The use of the PHP server and JavaScript
were proposed by the developer, and the use of Oracle and the commercial active
directory were requirements from the acquirer. There were no fundamental changes to
system functionality. The possibility for requirements changes to the system were
accommodated in the software contract.
In this project, the acquirer involvement in managing risks was expected to avoid
blaming the developer, and to provide support and resources to mitigate risks. In risk
identification and analysis, the developer discussed differences in risk perceptions with
the acquirer. The aim of the discussion was to reconcile perceptions and support
development processes. However, both stakeholders did not provide the detail of the
approach used to reconcile differences in perceptions. The developer also expected that
the acquirer could understand proposed solutions and then could cooperate to manage
risks. Therefore, the responsibility of risks was on both stakeholders.
87
2. Case 2: 2AcqBank
The participant of case 2AcqBank is a Division of Information Technology of one of
the biggest banks in Indonesia, playing the role of an acquirer. The division has
experience contracting its system development needs to external parties. The
participant’s stakeholder organisation acts as a developer. It is an Indonesian division of
an American multinational technology and consulting corporation. The company offers
a range of integrated solutions, products and services. The company has operated in
Indonesia since 1937.
In this case, the 11 OTS-specific risks were investigated on customer data integration
and master data management systems. These systems need to comply with central bank
regulation, and to support core banking services and to understand customer profiles.
OTS software was proposed by an IT consultant and agreed by the acquirer. The system
was built using COBOL, Oracle, .net and commercial SOA middleware. The SOA
middleware vendor, who is an international-based IT consultant, developed the systems.
Requirements changes in the systems were accommodated through discussion and
negotiation with the developer.
There were different approaches and perceptions between both stakeholders to manage
risks. According to the acquirer, without risk management, there would be differences
in expected results between both stakeholders because the developer might not
understand the acquirer’s needs. In risk identification, brainstorming and discussion
were used; in risk analysis and prioritization, planning and review meeting were used.
Project managers of the acquirer and the developer were responsible to understand
differences in risk perceptions. User acceptance testing (UAT) and proof of concept
(POC) were used before implementation.
3. Case 3: 3AcqTelco
The participant of case 3AcqTelco was a telecommunication operator in Indonesia,
acting as an acquirer. The participant’s stakeholder organisation is a developer. It is a
contact center service and outsourcing company, supported by a large Indonesian
telecommunication operator. The company was established in 1975. The company has
734 employees and contact center operations in eight cities in Indonesia.
88
In this case, the 11 OTS-specific risks were investigated on a customer billing
recommender system. This system is critical to remind and support other units that
distribute and collect invoices. As there are customers that do not pay invoices if they
do not receive them, this system is needed to ensure high levels of invoice collection.
The developed system was added to an existing customer billing recommender system.
To anticipate changes to the acquirer's organisational structure and customer segments,
the system was developed using a parameterized design. The system was built by a
domestic software developer company. The developer also has responsibility to
maintain the system for 3 years. The system was built using Oracle, PHP, DHTMLX,
PHPMailer, FPDF, Asterisk, and jQuery. This OTS software was proposed by the
developer.
The acquirer stated the importance to involve its developer in risk management. The
acquirer required risk analysis for structuring requirements, project planning and
scheduling. To meet the requirements, development processes were monitored. Overall,
both stakeholders were responsible for risk management.
4. Case 4: 4AcqGovt
The participant of case 4AcqGovt was an Indonesian governmental agency, acting as an
acquirer. The participant’s stakeholder organisation is a developer. It is an engineering-
based university spin-off company with 15 years of experience in information
technology-based projects. The project portfolio includes telecommunication systems,
avionics systems, multimedia and online marketing solutions, academics and library,
logistics and operational, e-Government, IT consultancies, and IT empowerment. The
developer's clients are government agents, corporations and organisations. The
developer has 50 university-educated employees.
The system was built using commercial business analytics and business intelligence
(BI) software, a commercial document management system, Oracle, MySQL Enterprise
database and PHP. This OTS software was the result of an independent study to select
89
OTS software using analytic hierarchy process (AHP), to select OTS software.
Requirements changes were anticipated by the acquirer because of the dynamic nature
of the acquirer’s organisation.
The acquirer pointed out the importance of anticipating and managing risks of
development, and focusing and prioritising tasks. Even though there was no formal
analysis of differences in risk perceptions, both stakeholders agreed on outputs of
project tasks, which were the results of scheduled meeting to reconcile perceptions. In
case 4AcqGovt, the acquirer was responsible for risk management.
Table 6.4 Summary of case study participants’ project and participant information
Participant in case
1Dev 2AcqBank 3AcqTelco 4AcqGovt
Project information
Number of teams (full time) 4 15 4 20
Number of teams (part time) 2 3 1
Percentage of a project member 50% 80% 40% 50%
who had the most previous
experience with general OTS-
based development
Percentage of the application 40-60 % 30-40% 45% 50%
functionality in the whole
system was provided by selected
OTS components
Percentage breakdown of couple level between OTS software and system [47]
Configuration 50% 10% - 10%
Integration 30% 40% 30% 20%
Customization 15% 40% 70% 20%
Development 5% 10% - 50%
Participant information
Position in the project Project Software Software Project
manager developer architect coordinator
and system
analyst
Position in the current Chief Internal Officer Chief
organisation Operational software Information
Officer developer Officer
Experience in software 7 5 2 20
development (year)
Experience in using OTS 5 4 1 22
components (year)
Experience using OTS 5-7 5 2 >30
components (in projects)
Educational background Bachelor Bachelor Bachelor Master
(degree) (continued
90
to Master)
Educational background (major) Computer Computer Computer Computer
Science Science Science Science
(Bachelor)
In the first data collection, the four participants were sent a questionnaire. The questions
are shown in Section 6.3.2. The participants of this study and their stakeholder then
provided inputs to the framework (these are shown in Appendix D).
In the second data collection, semi-structured interviews were used to collect data. The
interviews were conducted in each participant’s office in Indonesia. In the interviews,
all participants were asked questions by the researcher. The researcher took notes and
recorded using audio digital recorder. In the case 2AcqBank, the stakeholder of the
participant was also interviewed.
91
agreement process in the interview. However, all participants, including the
participant’s stakeholder in case 2AcqBank, finally agreed on the framework’s outputs.
After the interview, the framework outputs were sent to the stakeholder of case 1Dev,
3AcqTelco and 4AcqGovt for their agreement (the stakeholder of case 2AcqBank gave
its agreement in the interview). The stakeholder of case 1Dev, 3AcqTelco and
4AcqGovt also agreed on the framework outputs. In conclusion, the participants and
their stakeholders did not meet as a group during this study. Rather, their individual
responses independently agreed with each other.
The participants were asked to evaluate the framework using evaluation criteria for risk
assessment [133][134]. The participants were also asked about existing risks
management practices involving their stakeholders, and about suggestions and
improvements to the framework. Finally, the researcher also asked about comparison
between the framework and the method used in previous survey (in Chapter 5).
After the interview, the validation processes were performed by sending transcripts of
the interview and the questionnaire to all participants (not to the stakeholders of the
participants) to get feedback and confirmation. The transcripts included results of this
case study (will be detailed in Section 6.4). All participants confirmed the accuracy of
the transcripts.
Another form of data collection in the case study was inspection of documents and the
participants’ websites. Three participants sent their software contract-related documents.
Additional information was also accessed from the stakeholder’s websites. The
document provided information about contract and project management. The websites
provided information about stakeholder organisation and project description. These data
collection support data triangulation, which can increase construct validity [86], and
provide evidence of case study context.
This study asked the participant to only provide inputs to the framework and answer
questions in the first and second stages of data collection. This study did not require the
participants to learn and to use the framework [131].
92
6.3.7 Data Analysis
Non-inputs of the framework’s data (participant and project information, existing risk
management practices and analysing differences in risk perceptions) were analysed by
linking to theoretical propositions and by developing a case description [86].
An initial evaluation of the framework was measured using evaluation criteria for risk
assessment [133][134]. This study only used four criteria for evaluating the framework
(adaptability, feasibility, completeness and validity) [133][134], as described in Section
6.3.2. Existing risk management practices and suggestions from all participants were
then used to improve the framework [129] (provided in Section 6.5.4).
6.4 Result
In this section, the results are organised into three parts: the results of risk perspectives
from each case study participant and their stakeholders, the risk assessment evaluation
results, and the questionnaire and interviews results (data collection 1 and 2).
93
the framework are provided in the Appendix D. The next sub-sections details the risk
perceptions of each participant and their stakeholder.
94
Table 6.5 Framework outputs (stakeholder involvement and risk priorities) and agreement of risk responsibilities
Number of risk
Case
responsibility
1Dev 2AcqBank 3AcqTelco 4AcqGovt Dev Acq Both
Risk Stakeholder Risk Stakeholder Risk Stakeholder Risk
Stakeholder
resp. involvement resp. involvement resp. involvement resp.
involvement
Risk
Selection effort Dev: Collaborate Dev Dev: Both Dev: Both Dev: Both 1 3
ill-estimated Acq: Inform Collaborate Collaborate Collaborate
Acq: Acq: Acq:
Collaborate Collaborate Collaborate
Not adaptable Dev: Consult (*) Acq Dev: Both Dev: Dev Dev: Both 1 1 2
to requirement Acq: Collaborate Collaborate Collaborate Collaborate
changes Acq: Acq: Inform Acq:
Collaborate Collaborate
Requirements Dev: Collaborate Both Dev: Inform Acq Dev: Both Dev: Both 1 3
not negotiable Acq: Collaborate Acq: Collaborate Collaborate
Collaborate Acq: Acq: Involve
Collaborate (*)
Complicated Dev: Consult Acq Dev: Consult Both Dev: Inform Acq Dev: Involve Dev 1 2 1
multi-OTS Acq: Collaborate Acq: Consult Acq: Acq: Inform
components Collaborate
arrangement
Insufficient Dev: Collaborate Both Dev: Dev Dev: Dev Dev: Both 2 2
OTS (*) Collaborate Collaborate Collaborate
component Acq: Collaborate Acq: Consult Acq: Inform Acq: Involve
documents (*)
Lack of OTS- Dev: Collaborate Both Dev: Both Dev: Both Dev: Involve Both 4
driven Acq: Collaborate Collaborate Collaborate Acq: Involve
95
requirements Acq: Acq:
engineering Collaborate Collaborate
process
Maintenance Dev: Involve (*) Dev Dev: Inform Both Dev: Dev Dev: Involve Both 2 2
planning Acq: Consult Acq: Inform Collaborate Acq: Involve
unfeasible Acq: Inform
(*)
Upgrade Dev: Inform Acq Dev: Inform Both Dev: Inform Both Dev: Both 1 3
unfeasible Acq: Involve Acq: Inform Acq: Inform Collaborate
Acq: Involve
(*)
Lack of Dev: Collaborate Both Dev: Both Dev: Both Dev: Dev 1 3
information on Acq: Collaborate Collaborate Collaborate Collaborate
vendor Acq: Acq: Acq: Inform
Collaborate Collaborate
Lack of support Dev: Consult (*) Acq Dev: Dev Dev: Both Dev: Both 1 1 2
Acq: Involve Collaborate Collaborate Collaborate
Acq: Inform Acq: Acq:
Collaborate Collaborate
Reduced Dev: Consult Acq Dev: Inform Both Dev: Both Dev: Involve Dev 1 1 2
control of Acq: Collaborate Acq: Inform Collaborate Acq:
future Acq: Consult (*)
evolution of the Collaborate
system
Legend: Acq: acquirer; Dev: developer; Risk resp.: Risk responsibility; (*): either level of risk control or impact or both was changed before the
agreement of the proposed risk responsibilities during the interview
96
6.4.1.1 Risk Perceptions of Case 1Dev
In case 1Dev, the participant was a software developer and the stakeholder of the
participant was a software acquirer (see Table 6.3).
For the risk of selection effort ill-estimated, the developer was assigned to choose two
OTS software: a commercial PHP server and JavaScript. The other two pieces of OTS
software were defined by the acquirer: Oracle and commercial active directory. The
developer perceived itself to have high risk control and be most impacted by this risk.
The acquirer decided to use the PHP server, which complied with the acquirer’s IT
organisation procedure, and bought the license of PHP server. The acquirer already had
the license of oracle and active directory. The acquirer perceived itself to had low risk
control and impact (of the four pieces of OTS software used).
Consistent with the previous risk, the developer had high potential impact from the risk
of not adaptable to requirement changes, because the developer proposed the PHP
server. The developer had low control over the use of commercial OTS software. The
acquirer was perceived to be most impacted by this risk because the developed system
was used to service customers (critical application). Based on the high potential impact
to acquirer’s business, the acquirer perceived itself to have high risk control by
contracting the developer to build the system.
Both the developer and acquirer had high control and impact over the risk of
requirements not negotiable and lack of information on vendors (commercial OTS
software vendors).
The developer had limited control over the commercial OTS software because the
license of the commercial OTS software was owned by the acquirer. The acquirer was
perceived to be most impacted by the risk of complicated multi-OTS components
arrangement. There were two reasons for high risk impact. Firstly, multi-OTS software
is difficult to maintain. Secondly, from an organisation-wide audit process perspective,
multi-OTS software is difficult to audit. Therefore, the acquirer used higher control to
reduce the impact of this risk.
For the risk of insufficient OTS component documentation, the developer argued that the
risk responsibility should be shared by both parties because the PHP server was
97
proposed by the developer and then agreed to be used by the acquirer. To control for the
lack of the PHP server documentation, the developer customised the PHP server. The
acquirer argued that the developer should submit user manual documentation. Both
stakeholders perceived themselves to have high risk control and impact.
The developer used an estimation-based approach to select the PHP server. Therefore,
the developer had high risk control for the risk of lack of OTS-driven requirements
engineering process. In addition, the developer perceived itself to have high risk impact.
The acquirer also perceived itself to have high risk control and impact.
Even though there was a new release of any OTS software, the decision to upgrade the
OTS software in the system was the developers’. The new release of OTS software
might not have been compatible with other used OTS software. Therefore, risk impact
was low for the risk of maintenance planning unfeasible because the developer could
control the risk (high risk control) by deciding to use existing stable OTS software.
However, the acquirer perceived itself to have high risk impact because it was difficult
for it to maintain the developed system. On the other hand, the acquirer perceived itself
to have low control on OTS software from the commercial OTS software vendors.
For the risk of upgrade unfeasible, the developer perceived itself to have low risk
control and impact. On the other hand, the acquirer had control over the developer to
maintain the developed system but perceived itself to have a low risk impact.
Because the license of commercial OTS software belonged to the acquirer, the
developer perceived itself to have low risk control for the risk of lack of support. The
developer perceived itself to have high impact on this risk due to lack of technical
support and user training. However, having a license made the acquirer able to
negotiate with commercial OTS software vendors, therefore the acquirer perceived itself
to have high risk control. The acquirer perceived itself to have low risk impact because
the acquirer believed that a better understanding of OTS software features/capabilities
could substitute lack of support from vendor.
For the risk of reduced control of future evolution of the system, the developer claimed
that evolving system was hard to control (low control) and perceived this to have a high
98
risk impact. The acquirer argued that the risk was important to control by contracting
the developer (high control) and also perceived it to have a high risk impact.
Both the acquirer and developer perceived themselves to have high control and impact
for the risk of selection effort ill-estimated, lack of information on vendor (commercial
OTS software vendors) and not adaptable to requirement changes. For the last risk,
based on the software contract, both stakeholders agreed that requirement changes must
be accommodated.
According to the acquirer, the control and impact of the risk of requirements not
negotiable was high. The acquirer argued that the developed system must meet business
process requirements (high impact) and budget. To control the risk, because of limited
budget, the acquirer had an option to replace the developer if it could not meet the
requirements. However, the developer perceived this risk as to be under low control and
to have low impact.
Both the acquirer and developer perceived themselves to have high impact for the risk
of complicated multi-OTS components arrangement due to high cost for maintenance.
However, both perceived themselves to have low control for this risk.
For the risk of insufficient OTS component documentation, the acquirer believed that the
documents were needed (high impact). However, the acquirer found it impossible to
completely examine the documents (low control). For this risk, the developer had high
risk control and impact because the documents were needed and able to be accessed by
the developer.
Both the acquirer and developer perceived themselves to have high control and impact
for the risk of lack of OTS-driven requirements engineering process. The acquirer
maintained high control of requirement engineering process by executing a rigorous
proof of concept (POC) process. This started by tender, and then used a product demo
and proof of concept.
99
Both the acquirer and developer perceived themselves to have low control and impact
for the risk of maintenance planning unfeasible and upgrade unfeasible. For the second
risk, based on the acquirer and developer responses, there would be fewer problems to
upgrade the developed system because the upgraded system would be tested.
For the risk of lack of support, the acquirer perceived this as low risk control and impact
but the developer perceived it as high risk control and impact. The acquirer had low
impact of this risk because all possible problems would be handed to commercial OTS
software vendors. The developer also believed that the support should be delivered by
the commercial OTS software vendors or its official representatives.
For the risk of reduced control of future evolution of the system, both the acquirer and
developer perceived themselves to have low risk control and impact. The acquirer did
not require the latest release of OTS software as long as the system functioned.
However, if the latest version of OTS software could fix important bugs then the OTS
software would be updated.
For the risk of selection effort ill-estimated, both stakeholders perceived themselves
high risk control and impact. The control of this risk was revealed by the following OTS
selection processes. Firstly, the acquirer described a list of requirements. Secondly, the
developer offered OTS software to be used. Then, the OTS software was tested by the
acquirer using benefit cost analysis of the OTS software’s features. The acquirer also
compared the use of the OTS software by existing developer’s customers, which were
in auto finance, bank, airline and automotive industries.
For the risk of not adaptable to requirement changes, the acquirer perceived itself to
have low risk control and impact. The specification of requirements was defined by the
acquirer and was stable (detailed in the risk of maintenance planning unfeasible). Based
100
on the contract, the acquirer had the low risk impact because the developer had a
responsibility to meet the requirements. However, the developer perceived itself to have
high risk control and impact.
Both the acquirer and developer had high control and impact for the risk of
requirements not negotiable. The acquirer used a software contract to control this risk.
Requirement specification and OTS software features were documented in the software
contract. The contract included shared and individual roles. The contract was
negotiable, if needed changes were not set previously.
The acquirer perceived the risk of insufficient OTS component documentation as having
low risk control and impact but the developer perceived this risk as having high control
and impact. The acquirer expected knowledge transfer, e.g. user training and user
manual.
Both the acquirer and developer had high control and impact for the risk of lack of
OTS-driven requirements engineering process. Candidates for OTS software were
derived from an IT architecture from the acquirer’s organisation (predefined operating
systems, web servers and programming languages). Therefore, the list of requirements
described proposed OTS software, including the use of a trusted engine (see the risk of
selection effort ill-estimated).
For the risk of maintenance planning unfeasible, the acquirer perceived the risk as
having low control and impact because the contract, which lasted 3 years, did not
prioritise the use of the latest release of OTS software. To anticipate changes to the
101
acquirer's organisation structure and customer segments, the system was developed
using a parameterised design (for example, setting configuration files). The developer
perceived this risk as having high control and impact, which may relate to the long term
of the maintenance.
Both the acquirer and developer had low control and impact for the risk of upgrade
unfeasible. From the acquirer’s perspective, this risk related to the risk of maintenance
planning unfeasible. The developed system was tested using data from customer growth
over the last 5 years. As this risk related to the previous risk, the acquirer expected the
developer to have high risk control and impact.
Both the acquirer and developer had high control and impact for the risk of lack of
information on vendor (commercial OTS software vendors). The acquirer perceived a
high impact for this risk because the developed system was critical to support national-
wide operation. The acquirer had high control through benchmarking the use of OTS
software in the OTS selection. The quality of OTS software was a major factor in its
selection.
For the risk of lack of support and reduced control of future evolution of the system,
both the acquirer and developer perceived themselves to have high control and impact.
For the first risk, the acquirer perceived itself to have high risk impact because the
acquirer not only needed support but also related documentation. The acquirer asked the
developer to provide a new document for the changed system creating high risk control.
For the second risk, the high risk control for the acquirer was shown by the use of a
parameterised design to accommodate future changes.
According to the acquirer and developer, the control and impact of the risk of selection
effort ill-estimated was high. The acquirer employed an independent consultant to
conduct a study of OTS software selection using analytic hierarchy process (AHP). The
102
results of this study were then used to define features and functionalities of the
developed system. Thus, the acquirer perceived OTS software selection-related risk as
having high impact.
Both the acquirer and developer though that the risk of not adaptable to requirement
changes had high control and impact. The acquirer expected that all OTS software used
would still give optimal benefits if requirements changed.
The acquirer had low impact and high control for the risk of requirements not
negotiable. Based on the contract, requirements and development were the developer’s
responsibility (low risk impact for the acquirer). The acquirer perceived itself to have
high risk control because the acquirer regarded the terms of reference (TOR) as a
control. The acquirer also assumed that if the developer agreed with TOR then the
developer should understand system requirements. The developer perceived itself to
have high risk control and impact for this risk.
The acquirer perceived risk impact as low for the risk of complicated multi-OTS
components arrangement because the license arrangements of commercial OTS
software were not complex. In addition, the risk control for this risk was low because
the TOR managed multi-OTS components arrangement implicitly. According to the
acquirer, the developer could ask for additional OTS software licenses as long as stated
in the contract between the acquirer and commercial OTS software vendors. However,
the developer perceived itself to have low risk impact and high risk control for this risk.
The developer expected that the acquirer should have high risk control because the
acquirer had agreed on the proposed analysis and design of the developed system
(including all related risks). Therefore, both stakeholders were responsible.
For the risk of insufficient OTS component documentation, the acquirer believed itself to
have low impact and high control and the developer believed itself to have high control
and impact. The acquirer had high risk control because there was control over output
documentation for the developed system.
Both the acquirer and developer had low impact but high control for the risk of lack of
OTS-driven requirements engineering process. The acquirer thought if had low risk
impact because the development was contracted to an external party (the developer)
103
based on the TOR. The acquirer used the results of a technical study, informing the need
of integration of multi-OTS software, because a single platform was not sufficient (high
risk control). Therefore, system development needed modification to integrate multi-
OTS software to meet requirements.
Both stakeholders also perceived the risk of maintenance planning unfeasible to have
low impact but high control for themselves. The acquirer thought the risk had low
impact because the development was contracted to an external party (the developer),
based on the TOR. Future changes were anticipated by the acquirer because of the
dynamic nature of the acquirer’s organisation (high risk control).
For the risk of upgrade unfeasible, the acquirer perceived itself to have low impact and
high control, but the developer perceived itself to have high control and impact. The
acquirer regarded the terms of reference (TOR) as a control. Furthermore, if the
developed system did not meet requirements then, based on TOR, the acquirer would
not accept the developed system. However, the developer expected that the acquirer
should also be responsible for this risk because the acquirer was the party that ordered
the update to the developed system.
The acquirer indicated having low control and impact for the risk of lack of information
on vendor (commercial OTS software vendors). As system development was contracted
to the developer, the acquirer perceived this risk as having low impact. According to the
acquirer, the developer was very concerned about information available from
commercial OTS software vendors. Based on the TOR, the developer was responsible
to manage not only open source but also commercial OTS software used in the system.
On the other hand, the developer perceived this risk to have high control and impact.
Both stakeholders had high control and impact for the risk of lack of support. The
acquirer argued that lack of support would affect internal capacity building to maintain
and enhance the developed system. Internal capacity building was needed to reduce
vendor lock-in. In addition, the acquirer required that user training must be performed
by commercial OTS software representatives or certified person from commercial OTS
software vendors.
104
For the risk of reduced control of future evolution of the system, the acquirer perceived
itself to have high impact and low control. This risk had high potential impact because
of significant changes of policies. This risk had low control because the maintenance
was performed by the developer. On the other hand, the developer perceived the risk to
have low impact and high control.
Table 6.6. Comparison of evaluation criteria for risk assessment for four participants
105
6.4.3 Results of Data Collection 1 and 2
Table 6.7 summarises the questionnaire and interview answers, excluding answers to
questions about the framework inputs and evaluation. Complete results can be found in
Appendix E.
106
Table 6.7 Example of cross-case analysis for data collection 1 and 2, excluding framework inputs and evaluation criteria for risk assessment
Defined in Scope of Work. Defined in the project Acquirer organisation's Communication to achieve shared
contract. policy about understanding.
Risk outside responsibility stakeholder role and
of acquirer and developer responsibility on system Resource and capacity.
must be discussed with development.
respective stakeholder. Terms of reference.
Negotiation.
Recommendation Detailing Scope of Work Defined in software Describing level of Defining stakeholder responsibility
to improve risk (after using the contract. each stakeholder and clearly.
responsibility framework) shared responsibility.
Shared understanding of the definition of
Adding cost/benefit control.
analysis and
comparison. Detailed communication instruments,
e.g. meeting and Terms of reference
(ToR).
108
6.5 Discussion
This section discusses RQ3.3 and RQ3.4.
6.5.1 RQ3.3: How are OTS-specific risks perceived by developers and acquirers?
As our study focused on the perceptions of risk control and impact, we aimed to identify
key findings emerging from the cross-case analysis. We analysed stakeholder
involvement in Table 6.5 to identify key findings about risk priority. Finally, we
investigated commonalities of risk priorities between both stakeholders.
Three results related to requirements and contracts. Firstly, acquirers considered that
developers had a responsibility to meet the requirements. Secondly, the acquirer
regarded the contract as a risk control mechanism. In the 4AcqGovt case, for the risk of
requirements not negotiable, it was found that requirements and development are the
developer’s responsibility, and that the acquirer would not accept a developed system
that did not meet requirements (the risk of upgrade unfeasible). Finally, the contract
109
was used to change and negotiate requirements. In the 2AcqBank case for the risk of not
adaptable to requirement changes, the contract was used to control for requirement
changes by accommodating those requirements changes. Both stakeholders agreed that
the developed system must meet system requirements. In addition, in the 3AcqTelco
case, for the risk of requirements not negotiable, the contract was negotiated to
accommodate changes needed and not previously set.
Two results related to OTS software selection. Firstly, from the acquirer perspective, an
independent study of OTS selection using AHP could define provided features and
functionalities of the system requirements (the risk of selection effort ill-estimated in
case 4AcqGovt). Secondly, from the developer perspective in case 1Dev, because the
acquirer had agreed to the proposed OTS software from the developer, the responsibility
was shared.
There is one result related to design. In the 4AcqGovt case, for the risk of complicated
multi-OTS components arrangement, the developer expected that after the acquirer had
agreed to the proposed analysis and design of the developed system (including all
related risks), the responsibility should be shared.
Maintenance could also be related to design and contract from the perspective of the
acquirer. Future changes, which might be caused by the dynamic nature of the
acquirer’s organisation, were anticipated by the acquirers (in the 3AcqTelco and
4AcqGovt cases). One of the strategies used to reduce control of future evolution of the
system was a parameterised design approach (case 3AcqTelco).
110
Three results are related to maintenance and contract. Firstly, from the perspective of
the acquirer in the 4AcqGovt case, to control the risk of reduced control of future
evolution of the system, the developer performed system maintenance. Secondly,
according to the acquirer, based on the contract, the responsibility of vendor-related
information (such as vendor support and customer service) was on the developer (the
risk of lack of information on vendor). Finally, because the license of the OTS software
belonged to the acquirer, the developer perceived themselves to have high impact due to
lack of technical support and user training (in case 1Dev for the risk of lack of support).
When acquirers acquire a critical system, they tried to reduce risks through their
contract with developers. From the acquirers’ perspective, there were two risks
associated with contracting the developer: complicated multi-OTS components
arrangement in the 3AcqTelco case and not adaptable to requirement changes in the
1Dev case. As stated in the contract in the 3AcqTelco case, problems occurring during
the contract period must be managed by the developer, including OTS software vendor-
related risks. By contracting the developer, the acquirer in the 1Dev case wanted to
reduce risk impact of requirement changes in a critical system.
Contract-related risks were also found in other categories. For example, based on
contract, the developer chose OTS software; therefore the developer had high impact as
a consequence of proposing and choosing OTS software (in case 1Dev for the risk of
selection effort ill-estimated). Furthermore, as stated by the acquirer in the 4AcqGovt
case, the responsibility for understanding OTS-driven requirements and maintenance
planning was on the developer’s.
As found in the 1Dev case, the developer and acquirer had different perception of
limited control over OTS software. The developer perceived themselves to have low
control over OTS software in the integration and selection phases and the acquirer
perceived themselves to have low control over OTS software in the maintenance phase.
As both stakeholders experienced high risk impact, a possible explanation for the
difference in limited control over OTS software could be attributed to the importance of
OTS software during each phase for each stakeholder. In the risk classification, the
limited control over OTS software was classified as vendor, as shown in Table 6.8.
111
Two acquires had the same perception about OTS software vendors regarding the risk
of lack of support. In the 2AcqBank case, the acquirer perceived themselves to have low
risk impact because all support-related problems would be handed to the OTS software
vendor. The acquirer in the 1Dev case argued that having a license of OTS software
made them able to negotiate with the OTS software vendor.
Regarding budget, two acquirers perceived themselves to have high risk control. For the
risk of requirements not negotiable, the acquirer in the 2AcqBank case would have
chosen another developer to suit the existing budget and business process requirements.
In the 4AcqGovt case, the acquirer expected the OTS software to meet their budget for
the optimal benefit using OTS software if the requirements changed.
112
Table 6.8 Classification of key results regarding risk control and impact
Category Case
1Dev 2AcqBank 3AcqTelco 4AcqGovt
Requirements and The contract was used Based on the contract, Requirement and
contract to control for the developer had a development were the
requirement changes responsibility to meet developer’s responsibility
(acquirer and the requirements (acquirer)
developer) (acquirer)
The acquirer would not
The contract might be accept the developed
negotiated to system that did not meet
accommodate needed requirements (acquirer)
changes not set
previously (acquirer) The responsibility for
understanding OTS-driven
requirements was on the
developer (acquirer)
OTS software selection Because the acquirer Using a method in OTS
agreed on proposed OTS software selection (i.e.
software then the AHP) could define
responsible should be provided features and
shared (developer) functionalities of the
system requirements
(acquirer)
Design Because the acquirer
agreed on proposed
analysis and design of the
developed system then the
responsible should be
113
shared (developer)
The responsibility of
information on vendor was
on the developer (acquirer)
Contract The impact of critical The impact of critical
system made the acquirer system made the
contracting the developer acquirer contracting the
114
(acquirer) developer (acquirer)
Contract and OTS The developer chose OTS
software selection software; therefore the
developer had high impact
as a consequence of
proposing and choosing
OTS software (developer)
Vendor Having the license of OTS The acquirer perceived
software made the acquirer to have low risk impact
to be able to negotiate with because all support-
OTS software vendor related problems would
(acquirer) be handled to OTS
software vendor
(acquirer)
Vendor, OTS software The developer perceived to
selection and integration have low control over OTS
software in integration and
selection phase (developer)
Vendor and maintenance The acquirer perceived to
have low control over OTS
software in maintenance
phase (acquirer)
Budget The acquirer would The acquirer expected OTS
choose other developer software to meet budget for
to suit with existing optimal benefit of the use
budget and business of OTS software if
process requirements requirements changed
(acquirer) (acquirer)
115
6.5.1.2 Cross-Case Analysis
Here we identify key findings emerging not only from each individual case but also
across cases [136]. As the focus of this study is to investigate risk perceptions using the
stakeholder analysis approach, the findings were based on the relationship between
control and impact of investigated risks. Firstly, similar relationships were generalised
to identify the findings not only within each case but also across cases [136]. Secondly,
the findings were selected from each category by comparing evidence either within each
case or across cases [136]. Sometimes several categories in Table 6.8 were merged or
synthesised to generalise a finding. There are 11 key findings on risk control and
impact.
Finding 1 (high risk impact and control): A high risk impact made acquirers apply
higher control over the risk. Finding 1 is synthesised from three categories:
maintenance, contract and budget (see Table 6.8). In the maintenance category, the type
of impact are the difficulty of maintaining and auditing multi-OTS software (in the
1Dev case), and the need to build internal capacity and decrease vendor lock-in (in the
4AcqGovt case). For the second impact (in the maintenance category), user training
must be performed by OTS software vendor representatives or certified persons from
the OTS software vendor. In the contract category, because of the impact of critical
application, the acquirer in the 1Dev and 3AcqTelco cases contracted the developer to
build the application. Thirdly, in the budget category, the type of impact is not meeting
the budget for the optimal use of OTS software if requirements change (in the
4AcqGovt case), and not meeting business process requirements and having a limited
budget (in the 2AcqBank case). For the second impact in the budget category, acquirers
could choose another developer to suit the existing budget and business process
requirements.
Finding 2 (OTS selection): An acquirer can use OTS selection results to gain higher
control when defining system requirements, which has high impact. In the 4AcqGovt
case, the OTS selection was performed by an independent study of OTS selection using
AHP.
Finding 5 (contract): To lower risk impact, acquirers may contract an external party.
The relationship between an acquirer and its external party must be governed by a
software contract. Support for this finding can be classified into two groups: low risk
impact and low risk control; and low risk impact and high risk control. There were three
instances of the first group (low risk control and low risk impact). All support-related
problems were handled to OTS software vendor in the 2AcqBank case. Secondly, based
on the contract, in the 3AcqTelco case, the acquirer had the low risk impact because the
developer had a responsibility to meet the requirements. In the 4AcqGovt case, the
responsibility of vendor-related information was the developer’s.
There were four instances of the second group (low risk impact but high risk control).
Firstly, as shown in the 4AcqGovt case, the developer should have responsibility on
requirement changes and development, just as with understanding requirements (in the
second instance), and maintenance (in the third instance). Finally, in the 4AcqGovt case,
the acquirer used the contract to define system acceptance criteria. The first group, low
risk control and low risk impact, shows how the acquirer can lower risk impact by
contracting external parties. For the second group, the acquirer had higher control
because the contract was used to control the developer to have responsibility on
requirement changes, development and maintenance.
117
consequence of proposing and choosing OTS software. Furthermore, using the contract
to control for requirement changes required both stakeholders to accommodate the
changes (in case 2AcqBank and 3AcqTelco).
Finding 9 (licensing OTS software): An acquirer with an OTS software license can
better negotiate with an OTS software vendor, as seen in the 1Dev case.
Finding 10 (limited control over OTS software): Stakeholders have different concerns
about their limited control over OTS software. Developers tend to have limited control
over OTS software in the OTS selection process and OTS integration, and acquirers
tend to have limited control over OTS software in maintenance, after the end of a
contract.
The findings above emerged from cross-case analysis, and indicate that the analysis
framework helps to compare risk perceptions, to better understand stakeholders’ risk
perceptions. The comparison facilitates the agreement of proposed framework outputs
and risk responsibilities between both stakeholders. Furthermore, the comparison of risk
perceptions also support a discussion between stakeholders by providing opinions,
further understanding [24] and expectations of each stakeholder. If both stakeholders
disagree with proposed framework outputs, including proposed risk responsibilities, or
other stakeholder’s perceptions, the discussion is useful to elicit the reason of this
disagreement [24].
For acquirers, there are three results. There are two risks for which the acquirers do not
have high risk priority: maintenance planning unfeasible (for which they have two low
and two medium) and upgrade unfeasible (for which they have two low and two
medium). There are different responses regarding the risk of maintenance planning
unfeasible. For example, in the 1Dev case, the acquirer had a high risk impact due to the
difficulty of maintaining OTS software, but had a low control over the OTS software
vendor for maintaining OTS software. In the 3AcqTelco case, the acquirer perceived
themselves to have low risk control and impact because of not prioritising the latest
version of OTS software, and because of using a parameterised design. The acquirer in
the 3AcqTelco case for the risk of upgrade unfeasible had the same (low) level of risk
control and impact, and also the same reason as the risk of maintenance planning
unfeasible. The reason was the developed system did not prioritise the use of the latest
release of OTS software. However, the acquirer in the 2AcqBank case for the risk of
upgrade unfeasible had the same (low) level of risk control and impact, but for a
different reason, which was that upgrade would be feasible due to implementing testing
and bug fixing. The last result from the acquirers’ perspective is that they do not have
low risk priority (instead, they have three high and one medium) for the risk of
requirements not negotiable. In the 4AcqGovt case, the acquirer believed that software
requirements and development are the developer’s responsibility, as the acquirer
119
contracted the developer. The acquirer perceived this to be a high level of control. Thus,
the acquirer assumed that requirements-related risks did not give high impact to them.
Both stakeholders did not have a low risk priority for the risk of lack of OTS-driven
requirements engineering process. There are two examples showing that both the
developer (in the 1Dev case) and the acquirer (in the 4AcqGovt case) applied the OTS-
selection process in requirements engineering.
After summarising the above ten key results, there are six additional findings identified
from this study:
Finding 12: Developers have high priority for the risk of selection effort ill-estimated,
insufficient OTS component documents and lack of information on vendor.
Finding 13: Developers do not have low risk priority for the risks of not adaptable to
requirements changes and lack of support.
Finding 14: Developers do not have high risk priority for the risk of complicated multi-
OTS components arrangement.
Finding 15: Acquirers do not have high risk priority for the risk of maintenance
planning and upgrade unfeasible.
Finding 16: Acquirers do not have low risk priority for the risk of requirements not
negotiable.
Finding 17: Both stakeholders do not have low risk priority for the risk of lack of OTS-
driven requirements engineering process.
120
No formal risk management approach was shared by stakeholders.
Different techniques were in use to identify and analyse risk (common risk
identification techniques included brainstorming and discussion).
Different techniques were used to prioritise risks.
Stakeholders were not always able to predict other stakeholders’ perceptions.
This study proposes a framework to collect, analyse, compare and discuss differences in
risk control and impact perceptions between the participants and their stakeholders. The
framework outputs are stakeholder involvements and risk priorities for each
stakeholder. Risk responsibility is also proposed to belong to the stakeholder who has
higher risk control. This is because high influence and power over risk is needed to
control key decisions and task implementation [31][32][33], to mitigate risk. The
responsibility should be assigned to a stakeholder who has the competence and capacity
needed to discharge the responsibility [34][35]. However, if both stakeholders have the
same level of risk control, the risk responsibility is assigned to both stakeholders. These
outputs and risk responsibilities are then proposed to a developer and acquirer for
agreement. If both stakeholders can agree on the outputs and risk responsibilities, the
risk responsibilities are assigned as agreed; otherwise, both stakeholders can discuss the
proposed risk responsibilities by considering the analysis. However, although the
framework was designed to be used by a developer and acquirer, to avoid the effects of
learning to use [131] the framework, the case study participants were only asked to
complete the framework inputs and to agree on the framework outputs and proposed
risk responsibilities.
Table 6.5 shows that all participants and their stakeholders agreed on proposed risk
responsibilities and risk priorities. In discussing the agreement of proposed framework
outputs and risk responsibilities, there was initially one participant stakeholder (in the
4AcqGovt case) who did not agree with the five proposed risk responsibilities, but this
participant stakeholder finally agreed after discussing the process of determining risk
responsibility in the framework with the researcher.
This study found that the sources of risk responsibility are software project contract,
knowledge of risk, negotiation, communication, organisation policy, and resource and
capacity. Project contracts, knowledge on risk, and organisation policy can be classified
121
as formal controls, while negotiation and communication can be classified as informal
controls [84][82]. Knowledge can complement control [84], and competence and
knowledge affect the choice of controls [82], while resource and capacity were needed to
control risks. Therefore, the framework supports responsibility model [81][78][34][35],
which assigns responsibilities to the stakeholder who has competence and capacity
needed to discharge the responsibility. Defining responsibilities could help identify roles
[81][78] [34][35] to mitigate risks.
The results show that the responsibilities of the 11 risks vary as summarised in Table
6.5. The developers were assigned to have risk responsibilities in eight of the risks, the
acquirers in six of the risks, and both stakeholders in all 11 risks. See Table 6.5 under
the heading of Number of Risk Responsibilities for the risk of lack of OTS-driven
requirements engineering process, all participants and their stakeholders agreed on
proposed risk responsibility, which is ‘both stakeholders’. However, they disagreed for
other risks. One of the possible explanations why all participants and their stakeholders
did not agree is because stakeholder perceptions vary based on individual's or
organisation's background, experience, need and expectation [18][19]. This may also be
why participants were not able to predict other stakeholders’ perceptions.
Two of the eight risks (insufficient OTS component documents and maintenance
planning unfeasible) for which the developers had responsibility, the risk
responsibilities are in the developers and both stakeholders. Documentation is more
needed by the developers to integrate OTS components and also to maintain the system.
However, the acquirers have more responsibility for the risk of complicated multi-OTS
components arrangement than the developers. A possible explanation is because the
acquirers owned the license for OTS software.
Three participants changed their input of risk perception (provided in Section 6.4.1). As
mentioned in the data collection section, the changes of the framework’s inputs, risk
control and/or impact, happened before the agreement process in the interview. There
are two possible explanations for the changes. Firstly, the participants in case 1Dev,
3AcqTelco and 4AcqGovt got more understanding of the changed inputs during the
agreement process because the participants could confirm their understanding about
risks under study with the researcher during the interview. Secondly, the participants in
case 1Dev, 3AcqTelco and 4AcqGovt 4 were affected by their stakeholder answers of
122
the framework’s inputs. The first type of changes were data-driven and the second type
of changes either data-driven or expectation-driven [138].
If a stakeholder does not agree with other stakeholders’ perception on the proposed
framework outputs, the disagreement should be discussed (step 5 of the framework).
This discussion builds consensus and provides shared understanding, and is in line with
a study comparing software architectures [24] and risk management [22][23]. In sum,
the framework provides structured and methodological steps to define a risk and role
responsibility by comparing risk perceptions of both stakeholders.
According to the participants, the framework could also cover various and different
risks because the framework provided an input template and did not pre-define risks to
be analysed. The list of risks in the questionnaire could be adjusted. All participants
indicated that the framework could represent real risks; half of the participants reported
that the framework could be used to compare stakeholder perspective and be used to
propose risk responsibilities. Overall, all participants agreed that the framework outputs
could be used to manage risks, and help to inform scope of work (SoW) about
responsibilities of process-related risks.
123
6.5.4 Improving the Proposed Framework
Participants proposed improvements by comparing existing risk management practices
(found in this case study) with the framework, and making use of suggestions from all
participants. Three main improvements were proposed as follows.
1. Stakeholders should use the framework.
All stakeholders should be involved in assigning risk responsibilities (three cases).
The other case’s participant pointed out that the framework should be initiated by
the software acquirer, but that the framework would be more effective if it was
initiated by someone with the personal strength to reconcile differences in risk
perceptions [20].
2. Stakeholders should compare risks using the framework. The outputs proposed were
stakeholder involvement and risk priority (including risk responsibilities).
Three framework improvements were suggested:
Risks should be analysed at the beginning of a project
There is a need for a shared understanding of the definition of control and
impact. Therefore, the framework needs an introduction, a glossary of used
terms (such as control and impact) and related context, verification of
stakeholder’s answers, and a session to discuss differences of understanding of
definition of control and impact. The introduction and verification are already in
the steps of the framework.
The framework could include other risk attributes (such as cost/benefit) and risk
quantification for more detailed risk assessment.
3. The framework could be used to detail the level of stakeholder responsibility in
software contract and the scope of work (SoW).
The framework could also be used to clarify understanding of each stakeholder’s
roles and responsibilities in the ToR. Based on the participants’ opinions given
above, the framework has potential use for reconciling different process
perspectives by defining roles for processes in OTS-based custom software projects.
This study validates the potential uses and benefits of the framework. The aims of
analysing risks are to develop shared understanding, shared responsibility, stakeholder
commitment and shared perspectives; to avoid conflict; and to focus and to prioritise
tasks. Furthermore, the framework provides early understanding of risk perceptions,
124
facilitating the planning of a software project and formulating the software contract.
Overall, all participants agree that the framework can be used to initiate and support the
discussion to reconcile different risk and process perspectives between stakeholders.
Based on our case study results, compared with standard clauses on a software contract
[139] or existing organisational policies, stakeholders were expected to clarify
understanding of each stakeholder’s roles and responsibilities in the terms of reference
(ToR) (see Recommendation to improve risk responsibility from participant
4AcqGovet in Table 6.7). It was assumed that the tender winner had understood the
ToR. However, the tender winner tended to be overoptimistic and the winner did not
discuss who would control and be impacted by risks. Thus, the framework provides a
bi-directional approach to discuss and negotiate risk responsibility.
A summary of the case study evidence and related propositions and literature related to
evaluation, and improvement of the framework is presented in Table 6.9
125
contract, knowledge on risk, negotiation, [84][82]
communication, organisation policy, and resource and
capacity (Q2.2.4)
The use of the framework to detail scope of work and Improvement for proposition
software contract, detail of stakeholder and shared P1, P2, and solution for P3
responsibility (two participant), shared understanding
of the definition of control, adding required attribute to
analyse and clarification the understanding of each
stakeholder’s roles and responsibilities on (ToR)
(Q2.2.5)
Risk expectation driven by risk perception (two This finding supports
participants) (Q2.2.6) [18][138]
Understanding risk control, impact and responsibility. -
Adding required attribute to analyse (Q2.2.6)
Integration of the framework in the beginning of the Improvement for proposition
software project (three participants) (Q2.2.7) P1, P2, and solution for P3
Inability to predict other stakeholder perceptions Stakeholder perceptions vary
(Q2.2.8.a) [18]
Comparison of actual risk perceptions better than Objectively reviewed decision
perception-based risk (two participants). Structured, [140]
methodical and measured approach (Q2.2.8.b)
126
related to the first risk. The participant in case 3AcqTelco asked the researcher, “Why
does the risk of maintenance planning being unfeasible have high risk control and
impact, but the risk of upgrade being unfeasible have low risk control and impact?”).
The participant believed that there was a relation between these two risks. These
findings support propositions of relationship between perception and expectation
[18][138] (see Table 6.9). These findings also indicate that identifying stakeholders’
risk expectations can help to ensure risks are addressed [38].
127
In all cases, even though there were possibilities that inaccurate answers were provided
to meet the expectations of others, these answers were minimised in this research. First,
one of the benefits using a completed project in each case is to encourage the
participants and their stakeholder to give inputs of the framework based on facts. Our
case study has found that there were no specific methods used to analyse differences in
risk perceptions. Therefore the use of retrospective risk perceptions in the participants'
completed project was expected not significantly influencing opinions given by the
respondents. Second, the participants and their stakeholder were only asked to provide
their risk control and impact (as inputs of the framework) and not others’ expected risk
control and impact. In addition, when each party provided its answers, it would not
know other's answers.
6.6.4 Reliability
To increase reliability [86], this study followed existing case study guidelines [86]
[131][132][129]. This study also recorded and maintained a case study database to
increase reliability [86]. To simplify the presentation of the thesis, we decided to not
include quotes of the participants (there is one quote from the participant (Section
6.5.6).) and to describe and summarise the case study data and outcomes. To support
audit trail, we provide cross-case analysis from data collection 1 and 2 in Appendix E.
6.7 Summary
This chapter has investigated differences in perception of 11 risks of OTS-based custom
software projects between developers and acquirers by comparing their perceptions of
risk control and impact. It used a multi-case study method. Based on the method used to
analyse the survey data in Chapter 5, a risk framework was developed to collect,
compare and analyse these differences. The framework outputs are stakeholder
involvements and risk priorities for each stakeholder. There were four participants in the
128
case study: these were among 42 invited respondents of our survey (see Chapters 4 and
5), who agreed to participate and could also involve their stakeholder in this study.
This study yielded 17 detailed findings related to levels of risk control and impact, and
proposed risk priorities. As the participants were not able to predict other stakeholders’
perceptions, due to differences in individuals, their organisations’ background,
experience, need and expectation [18][19], these findings can serve as an initial
checklist for understanding other stakeholders’ risk perspectives.
Five findings about risk perceptions are specific to developers. Firstly, developers often
decide to use existing stable OTS versions to control the impact of system maintenance.
Secondly, developers without an OTS software license often lack support from OTS
software vendors.
There are three different findings regarding risk priority. Firstly, developers have high
priority for the risk of selection effort ill-estimated, insufficient OTS component
documents and lack of information on vendor. Secondly, developers do not have low
risk priority for the risks of not adaptable to requirements changes and lack of support.
Lastly, developers do not have high risk priority for the risk of complicated multi-OTS
components arrangement.
Eight findings about risk perceptions are specific to acquirers. The first is that risks with
high impact make acquirers apply higher control over these risks. Secondly, acquirers
can use OTS selection results to define system requirements. Thirdly, in maintenance,
acquirers can anticipate future changes by asking developers to use a parameterised
design (for example, setting configuration files). In addition, there are two findings
related to the contract: to lower risk impact, acquirers can contract an external party;
however acquirers will have lower control when they contracts developers and use OTS
software. With an OTS software license, acquirers can help negotiate with an OTS
software vendor. There are two findings related to risk priority. Acquirers have high risk
priority for the risk of requirements not negotiable. Lastly, acquirers have low risk
priority for the risk of maintenance planning and upgrade unfeasible.
The other four findings about risk perceptions relate to both stakeholders. Firstly, using
a software contract to control risk leads to consequences regarding what must be agreed
129
in the contract. Secondly, stakeholders have different concerns about their limited
control over OTS software: developers tend to have limited control over OTS software
in the OTS selection process and OTS integration, while acquirers tend to have limited
control over OTS software in maintenance, after the contract ends. Thirdly, both
stakeholders have risk responsibility if an acquirer agrees on OTS software selection,
and a developer’s proposed system design decision. Lastly, both stakeholders do not
have low risk priority for the risk of lack of OTS-driven requirements engineering
process.
Although these findings might not be very surprising and are not generalised beyond
OTS-based custom software projects and the 11 risks under study, the findings do
provide insight into developers’ and acquirers’ perceptions of risk control and impact.
The framework in this study added substantially to our understanding of how to provide
a methodical approach to analyse different process-related risk perceptions [16]. The
130
framework is expected to increase understanding of risk responsibility for both
stakeholders by discussing, negotiating [24] and clarifying risk responsibility in two
directions instead of deriving it from standard contract clauses [139] or existing
organisational policies. Moreover, the framework has a potential use for analysing
differences in process perspectives, by identifying stakeholders’ roles and their level of
involvement [36][37][13]. We also believe that the method used in this study could be
applied to non-OTS-based custom software projects, to analyse differences in process-
related risk perceptions.
Our study proposed a default position about which stakeholders should be responsible for
a risk based on which stakeholder has higher control and power of each risk [31][32][33].
Further investigation on other related factors influencing risk responsibility is strongly
recommended. To analyse differences in risk perception involving stakeholders other
than developers and acquirers, the framework should be adjusted, evaluated and
validated. There are another two limitations. Firstly, there was possibility of subjective
understanding of the concepts of risk control and impact by all participants and their
stakeholders. Secondly, one of four participants’ stakeholders did not give detailed
opinions about risk perspectives.
131
7 Discussion and Conclusion
This research has investigated OTS-based custom software project, and has identified
risks involved [16], and analysed differences in risk perceptions and proposed
stakeholder responsibility for risks [22]. This has been done, not only from developers’
perspective, but also from acquirers’ perspective. Firstly, we investigated the processes
involved in OTS-based software acquisition using a systematic mapping study. Then,
we used a second systematic mapping study to identify risks of OTS-based custom
software projects from developers’ and acquirers’ perspectives. Finally, we extended an
existing stakeholder analysis framework [25][12][26][14][13][10] to compare, analyse
and discuss [22][23][24] risk perceptions between developers and acquirers of OTS-based
custom software projects in a survey and a multi-case study.
This chapter has five sections. Firstly, it reviews the research contributions. Then this
research is compared with related work. The next section discusses implications for
research, including future work. The fourth section discusses implications for practice
and the last section identifies limitations. In the comparison with related work and
research contribution sections, we refer to the research questions:
132
7.1 Research Contributions
This thesis makes several findings and contributions to empirical software engineering
knowledge, as follows.
Although our mapping study results have shown that both the software acquisition
process standard and OTS-based software acquisition have the same overall process
lifecycle, there are process differences between them. The first difference is that in
OTS-based software acquisition, architecture and OTS selection criteria are defined and
adjusted iteratively in conjunction with software requirements specifications. Secondly,
there are some OTS adoption characteristics that must be considered when acquiring
OTS software (for example, the credibility of the decision maker and previous exposure
to acquisition decisions) [108]. Thirdly, managing relationships between OTS vendors
and developers can provide benefits in commercial negotiations with acquirers [109].
Based on the mapping study results, there are three processes that could be added into
(generic) software acquisition and OTS-based software acquisition: decision-making to
make or buy software, architectural decision-making during software acquisition, and
managing relationships between developers and acquirers. Based on the literature
review, there are three processes in common between acquirers and developers of OTS-
based software: make vs. buy decisions, OTS selection and vendor interaction. A better
understanding of OTS-based software acquisition and development processes, and of
differences in backgrounds, experiences, needs, expectations [18], goals, and structures
[19] should improve understanding of differences in process-related risk perceptions
between both stakeholders.
133
RQ2 contributions: We identified risks of OTS-based and custom software
development and acquisition.
We have identified and classified risks of OTS-based and custom-based software, not
only from developers’ perspective but also from acquirers’ perspective. These risks
have been classified into 17 categories: planning, requirements, design, integration and
testing, system lifecycle, maintenance, project closure, software, OTS products, cost,
environment, people, systems engineering, vendor relationships, project management,
contract and legal.
The results consist of specific and generic risks in OTS-based and custom software
acquisition and development. OTS-specific risks can be found in all risk categories. The
results show that most risks found in OTS-based software acquisition have the same
concerns as risks in OTS-based software development. In both custom software
acquisition and development, specific risks are related to contracted or outsourced
software development projects. Technical-related risks are found less often in
acquisition and project management-related risks are found less often in development.
There are generic risks, which are common to all software projects, in 13 risk categories
of OTS-based software development and 14 risk categories of OTS-based software
acquisition except in the risk category of OTS products. In this study, poor requirement
definition and analysis, found in the requirements category, is the only generic risk
found that was shared between OTS-based and custom-based software development and
acquisition.
RQ2 and RQ3 contributions: The survey and case study show that in OTS-based
custom software projects comparing and discussing risk perceptions about risk
control and impact between both stakeholders enhance our understanding of their
risk perceptions and differences in their risk perceptions.
In our survey and case study, we focused on 11 process-related risks that should be
relevant not only to developers but also to acquirers in OTS-based custom software
projects. There are four detailed findings about comparing and discussing risk
perceptions between stakeholders.
134
RQ2 contributions: There are different perceptions about risk occurrences in
OTS-based custom software projects between the developer and acquirer
perspectives (survey-based findings).
The survey partially confirms a previous study [27], and complements it with findings
about the acquirer perspective. The results of the survey focus on 11 process-related
risks, and show that more of these risks occurred more frequently in software
acquisition than in software development (nine vs. five risks out of these 11 risks; as
shown in Table 4.11). Six risks occur more frequently in software acquisition than in
software development: selection effort ill-estimated, not adaptable to requirement
changes, requirements not negotiable, maintenance planning unfeasible, upgrade
unfeasible and lack of information on vendor. Two risks occur more frequently in
software development than in software acquisition: insufficient OTS component
documents and a lack of vendor technical support and training. There are three risks
that occurred equally frequently in both software development and acquisition:
complicated multi-OTS component arrangement, lack of OTS-driven requirements
engineering process, and reduced control of future evolution of the system.
135
respondents’ agreement about which stakeholders have high control of each risk.
Responsibilities for all risks (except requirements not negotiable) can be proposed to
belong to the developer. With regard to requirements not negotiable, we proposed that
both stakeholders share responsibility for the risk. The proposed risk responsibility does
not consider all potentially-related factors to allocate risk responsibility, but the method
is consistent with responsibility models [81][78][34][35]. A responsibility model
describes responsibilities within a system under development, agents assigned to these
responsibilities and resources used to discharge these responsibilities [78]. In addition, a
responsibility should also be assigned to the stakeholder who has the competence and
capacity needed to discharge the responsibility [34][35]. Competence and knowledge
affect the choice of control [82], while resource and capacity were needed to perform
control over risks.
RQ3 contributions: We made 17 findings about level of risk control and impact,
and proposed risk priority (case study-based findings)
Based on the evidence from the cross-case analysis, it can be argued that each
stakeholder has their own and shared perceptions regarding the risks of OTS-based
custom software projects. The case study revealed 17 detailed findings related to levels
of risk control and impact, and proposed risk priorities.
There are five findings about risk perceptions specific to developers. Developers tend to
perceive risks related to OTS vendors (such as a lack of technical support and training,
a lack of vendor information, and OTS software documentation) as more important than
the use of multi-OTS components. Developers also perceive risks related to
requirements and OTS selection to be important. To control the impact of system
maintenance, developers can often decide to use an existing stable version of OTS
software.
Eight findings about risk perceptions are specific to acquirers. Generally, acquirers tend
to target higher control over a risk having high impact. Acquirers perceive risks related
to requirements as more important than risks related to maintenance and upgrade.
Acquirer can use OTS selection results to define system requirements. An acquirer can
lower a risk’s impact by contracting an external party. For example, all support-related
problems will be handled to an OTS software vendor. However, contracting a developer
and using OTS software will give acquirers less control over the risks. An acquirer can
136
negotiate with an OTS vendor to manage the use of and support for OTS software. To
anticipate future change, an acquirer can ask a developer to use a parameterised design
(for example, setting configuration files).
Four findings about risk perceptions relate to both stakeholders. Firstly, using a
software contract to control risks leads to consequences regarding what must be agreed
in the contract. As mentioned by a case study participant, its developer might not fully
understand a contract because they might only use it to win a project. Next, developers
and acquirers have different kinds of control over OTS software: developers tend to
have limited control over OTS software in the OTS selection process and OTS
integration because of not having the license of OTS software, while acquirers tend to
have limited control over OTS software in maintenance because of limited access and
knowledge over OTS vendor and software. Thirdly, there is a shared risk responsibility
for agreement on OTS software selection, and the developer’s proposed system design
decisions. Fourthly, both stakeholders have a high risk priority for the risk of lack of
OTS-driven requirements engineering process, which is related to how requirements
formulation is affected by the availability of OTS software and how requirements are
mapped to OTS software [77].
The above findings show that developers and acquirers have different concerns about
risk controls and responsibilities. Developers tend to focus on risks related to OTS
vendors, OTS software integration, OTS selection and requirements. Acquirers tend to
focus on risks related to requirements and OTS selection. A better understanding of
these different concerns can help stakeholders to analyse related risks.
RQ3 contributions: The case study findings enhance our understanding of the
importance of risks from developers’ and acquirers’ perspectives by taking
account of the level of risk control and impact.
The case study findings show that: the higher a risk control, the more important the risk;
for risks with lower control, the higher the impact, the more important the risk is (this is
further discussed in the next section).
137
RQ3 contributions: A proposed framework to analyse differences in risk
perceptions and to assign risk responsibilities between developers and acquirers.
We proposed a framework to analyse differences in risk perceptions and to assign risk
responsibilities between a developer and acquirer. The framework was initially
developed in our survey to analyse the survey data of differences in perceived risks
between developers and acquirers (see Chapter 5). The framework extends stakeholder
analysis [25][12][26][14][13][10] and can be used to compare, analyse and discuss level
of risk control and impact between a developer and acquirer. The framework proposes a
default position about which stakeholders should be responsible for a risk, based on which
stakeholder has higher control over the risk. In our case study, the framework was used to
facilitate a discussion between both stakeholders to assign a risk responsibility based on
the proposed risk responsibility.
Our findings show that all participants and their stakeholders agreed on proposed risk
responsibilities and risk priorities. As confirmed by the case study findings, a
responsibility should also be assigned to the stakeholder who has the competence and
capacity needed to discharge the responsibility [34][35] and should be agreed between
both stakeholders. In the case that stakeholders disagree with the proposed framework
outputs and risk responsibilities, or the other stakeholder’s perspective, the discussion is
useful to elicit the reason for this disagreement [24]. In the context of a specific project,
the framework provides an explicit recognition of different risk perceptions, to inform risk
management negotiation through dialogue, deliberation and communication [16]. The
framework could help stakeholders reduce ‘gut feeling’ judgments and to ensure that
decisions regarding the assignment of risk responsibilities are more objectively reviewed
[140].
The findings for RQ1 identify six OTS-based software acquisition processes: decision-
making to make or buy software, architectural decision-making, OTS selection,
managing relationships between developers and acquirers, managing relationships
between OTS adoption and acquirers’ organisations, and managing relationships
between OTS vendors and developers. Our case study results and findings provide
empirical evidence that these processes exist in practice. For example, the acquirer in
the 1Dev case performed OTS software selection. There are three processes in common
between OTS-based software acquisition and development processes: make vs. buy
decisions, OTS selection and vendor interaction. The responsibilities of these common
OTS-based software processes will vary depending on stakeholders’ goals and
structures [19].
Instead of using a purely technical approach [145][76], we focus on human and social
factors in risk management of OTS-based custom software projects. This research
investigates risks from human and social perspectives [146][147] by taking into
account developer and acquirer perspectives. To manage risks effectively, it is
important for all software project stakeholders to understand the risks [16]. To initially
139
answer RQ2, our study identified and classified risks of custom and OTS-based
software, from both perspectives. These results complement existing lists of software
project risks, which are from the perspectives of project managers [30][58][57][71][59]
and risk team members [22]. Our work is also in line with value-based software
engineering because it deals with different stakeholders’ perspectives to overcome the
limitations of a value-neutral approach [148][149]. An example of the value-neutral
approach is when every requirements, use case, object and defect is treated as equally
important [148][149]. In this study, perspectives of both stakeholders are investigated
because risks can affect both stakeholders [8].
This research supports previous works [8][22][23] arguing that, to manage risks, key
stakeholders must be involved from the beginning of software projects. However, there
are several differences between this research and these prior works. Firstly, compared
with Schmidt et al. [8], who argued to cooperatively manage risks, our research gives
detailed procedures to asses differences in risk perceptions between both stakeholders.
Secondly, prior research on control requires a ‘risk advisor’, whose tasks are to mediate
between both stakeholders and to manage risk [22][23]. In contrast, our risk framework
only includes developers and acquirers as the framework users, because a study by Keil
et al. [66] noted that an ‘outside consultant’ did not identify more risks than insiders nor
make more risk-averse decisions.
140
making risk responsibility decisions. The benefits of using structured discussion are that
stakeholders can better understand and learn about other stakeholders’ perceptions,
enabling them to make a more informed decision [24]. In requirements management, the
EasyWinWin approach is a requirement elicitation and negotiation methodology, which
supports expectation management, adopts prioritisation techniques and is supported
with groupware tools [150][149]. There are two tactics from EasyWinWin relevant for
our framework: structured procedures to handle conflicted stakeholder goals [150][149];
and the use of groupware to increase productivity of collaborative work [151]. Below
we discuss how our framework compares and analyses differences in risk perception of
OTS-based custom software projects between developers and acquirers for answering
RQ3.
In the case study (Table 6.5), four developers identified ‘collaborate’/high risk control and
impact for three risks (selection effort ill-estimated, insufficient OTS-component
documents and lack of information on vendor). It is interesting to compare these risks to
the related risks in the survey of risk perceptions in the previous chapter. These case study
results corroborate our survey results (detailed in Chapter 5), which showed that
developer respondents perceived that they could best control and were most impacted by
these risks. However, in the survey, the developer respondents perceived that for the risk
of selection effort ill-estimated, both stakeholders were most impacted. This result is
confirmed by our case study finding that when acquirers agree with developers on
proposed OTS software, then the responsibility should be shared (detailed in Section
6.5.1.1). Regarding the risk of there being insufficient OTS component documents, our
survey and case study findings are consistent with Mahmood and Khan’s [124] study
reporting that component documentation is needed when integrating OTS software. In
addition, from the developers’ point of view, our survey and case study findings support
the previous studies [74][27] that pointed out the importance of monitoring information
from OTS vendors regarding product upgrades and technical support, not only during
development but also during maintenance.
In our framework, the priority of a risk is based on the result of mapping it into a 2x2
control/impact matrix. Risk priority shows which risks are most important to address
[30]. Instead of using risk-exposure quantity [30] or multiplication on ordinal scale [38],
the framework prioritises risk using available information [38] by comparing values of
141
the mapping results from a developer and acquirer. In contrast with previous studies
[16][59][128] reporting that stakeholders tend to perceive the importance of certain risks
as higher than others if they cannot control the risks, our case study found that the higher
control over a risk, the more important the risk. However, our framework corroborates
previous studies [16][59][128] showing that for risks with lower control, the impact of the
risks directly affects the importance of the risks. In short, the higher the impact, the more
important the risk is.
Our findings show that analysing project stakeholders can be integrated into software
project risk management. Two examples are similar to our approach. Firstly, a Software
Development Impact Statement (SoDIS) associates every software development task
142
with involved stakeholders and then uses structured questions to assess the specific
project risks generated by that particular association [9]. Therefore, our framework
could provide better justification in cases when SoDIS might assign more than one
stakeholder to be responsible for a task-related risk. Secondly, the Outcome-Based
Stakeholder Risk Assessment Model (OBSRAM) provides guidance in identifying and
managing project risks arising from project stakeholders [10]. OBSRAM identifies
stakeholder influence and project's impacts on stakeholders, and then assesses and
prioritises stakeholder risks using risk scope, domain scope, risk severity and risk
probability. Compared to OBSRAM, in which risk prioritisation does not consider
stakeholder influence and project’s impacts on stakeholders, our framework prioritises
risks based on stakeholders’ risk control and impact.
There are similarities between our framework and two studies of role assignment in
software development: role-based software development (RBSD) [156] and the
Capabilities-Oriented Software Process Model [157][158]. The common steps between
our framework and RBSD are negotiation and assignment of responsibility. However,
RBSD uses more factors to assign roles (in our framework called responsibilities):
rights, responsibilities, accessibilities, and collaboration methods. The Capabilities-
Oriented Software Process Model assigns people to roles according to their capabilities,
and capabilities demanded by the roles [157][158]. In The Capabilities-Oriented
Software Process Model, the first step to assign a role starts by comparing a personal
and role profile. The next step is identifying the closest match between the personal and
role profiles. Another capability-based role assignment method uses Little-JIL to assign
roles based on available and matching capabilities to perform a task [159]. A
mathematically-based formal model integrating the Delphi method could help project
leaders to take into account as many factors as possible in assigning individuals to
project roles [160]. It is also interesting to consider the use of network theory in the
problem of task allocation for coordinating software development [161] in our future
work.
One of the aims for eliciting and reconciling differences in stakeholders’ perceptions is
to support expectation management [150][149]. Our study on RQ3 found that
participants had invalid expectations about each others’ risk perceptions. Therefore in
143
our future work, we would like to integrate our framework with stakeholders’
expectations, as in [150][149][38][64].
7.5 Limitations
Finally, a number of important limitations should be considered.
Even though open source software (OSS) may present different risks, the case study
found that the process-related risks were relevant to OSS. Therefore, we believe
that process-related risks in this research are common to COTS, OTS, component-
based software and OSS.
Although the framework was designed to be generically applicable to collect,
compare and analyse differences in risk perceptions, this capability has yet to be
implemented outside OTS-based custom software projects.
145
In comparing levels of risk control and impact (low and high), there was a
possibility of stakeholders’ subjective understanding of risk control and impact. To
avoid this in our case study, we clarified and asked for stakeholders’ agreement on
levels of risk control and impact (see Table 6.5).
This research analysed OTS-based custom software projects and differences in the
perception of 11 process-related risks relevant to the developers and acquirers. This
could limit generalisability of this research, because there are more risks in this
context. However, the framework used to analyse these risks is not a risk
identification technique, so that the framework can be used to analyse different
risks other than the 11 risks.
To analyse differences in risk perceptions involving stakeholders other than
developer and acquirer, the framework should be modified by adding the dimension
of risk control/impact matrix, and re-validated.
Having numerous differences in the four cases (Table 6.4), the case study
represented different contexts, which were expected to vary the sample of the
population under study. However, case studies have a limited ability to generalise
results to populations [86]; therefore, the case study should be replicated in
different cases.
146
Bibliography
[1] C. L. Braun, ‘A lifecycle process for the effective reuse of commercial off-the-
shelf (COTS) software’, in Proceedings of the 1999 Symposium on Software
reusability, Los Angeles, California, United States, 1999, pp. 29-36.
[2] J. Grudin, ‘Interactive systems: bridging the gaps between developers and users’,
Computer, vol. 24, no. 4, pp. 59-69, 1991.
[3] M. Keil and A. Tiwana, ‘Relative importance of evaluation criteria for enterprise
systems: a conjoint study’, Information Systems Journal, vol. 16, no. 3, pp. 237-
262, 2006.
[4] D. Carney, Assembling Large Systems from COTS Components: Opportunities,
Cautions, and Complexities. SEI Monographs on Use of Commercial Software in
Government Systems. Software Engineering Institute, Pittsburgh, USA, 1997.
[5] M. Torchiano and M. Morisio, ‘Overlooked aspects of COTS-based
development’, IEEE Software, vol. 21, no. 2, pp. 88-93, 2004.
[6] IEEE, ‘IEEE recommended practice for software acquisition’, IEEE Std 1062,
1998 Edition, 1998. .
[7] E. Rosendahl and T. Vullinghs, ‘Performing Initial Risk Assessments in
Software Acquisition Projects’, Software Quality — ECSQ 2002, 2002.
[8] C. Schmidt, P. Dart, L. Johnston, L. Sterling, and P. Thorne, ‘Disincentives for
communicating risk: a risk paradox’, Information and Software Technology, vol.
41, no. 7, pp. 403-411, May 1999.
[9] D. W. Gotterbarn and S. Rogerson, ‘Responsible risk analysis for software
development: creating the software development impact statement.’,
Communications of AIS, vol. 2005, no. 15, pp. 730-750, 2005.
[10] R. W. Woolridge, D. J. McManus, and J. E. Hale, ‘Stakeholder Risk Assessment:
An Outcome-Based Approach’, IEEE Software, vol. 24, no. 2, pp. 36-45, 2007.
[11] AS/NZS ISO 31000:2009 : Risk management - Principles and guidelines.
Standards Australia, 2009.
[12] R. E. Freeman, Strategic management: A stakeholder approach, vol. 1. Pitman,
1984.
[13] E. A. Whitley and A. Pouloudi, ‘Stakeholder identification in inter-
organizational systems: gaining insights for drug use management systems’,
European Journal of Information Systems, vol. 6, no. 1, pp. 1-14.
[14] J. Kaler, ‘Differentiating Stakeholder Theories’, Journal of Business Ethics, vol.
46, pp. 71-83, 2003.
[15] M. Jorgensen, ‘How to Avoid Selecting Bids Based on Overoptimistic Cost
Estimates’, IEEE Software, vol. 26, no. 3, pp. 79-84, 2009.
[16] M. Keil, A. Tiwana, and A. Bush, ‘Reconciling user and project manager
perceptions of IT project risk: a Delphi study1’, Information Systems Journal,
vol. 12, no. 2, pp. 103-119, 2002.
[17] L. Zhu, R. Jeffery, M. Staples, M. Huo, and T. Tran, ‘Effects of Architecture and
Technical Development Process on Micro-process’, in Software Process
Dynamics and Agility, vol. 4470, Q. Wang, D. Pfahl, and D. Raffo, Eds. Springer
Berlin / Heidelberg, 2007, pp. 49-60.
[18] S. Hornik, Houn-Gee Chen, G. Klein, and J. J. Jiang, ‘Communication skills of
IS providers: an expectation gap analysis from three stakeholder perspectives’,
147
IEEE Transactions on Professional Communication, vol. 46, no. 1, pp. 17- 34,
2003.
[19] R. Sabherwal, ‘The evolution of coordination in outsourced software
development projects: a comparison of client and vendor perspectives’,
Information and Organization, vol. 13, no. 3, pp. 153-202, Jul. 2003.
[20] S. Adolph, P. Kruchten, and W. Hall, ‘Reconciling perspectives: A grounded
theory of how people manage the process of software development’, Journal of
Systems and Software, vol. 85, no. 6, pp. 1269-1286, Jun. 2012.
[21] A. Bröckers, ‘Process-based software risk assessment’, in Software Process
Technology, W. Schäfer, Ed. Springer Berlin Heidelberg, 1995, pp. 9-29.
[22] F. J. Heemstra and R. J. Kusters, ‘Dealing with risk: a practical approach’,
Journal of Information Technology, vol. 11, no. 4, pp. 333-346, 1996.
[23] F. J. Heemstra, R. J. Kusters, and H. de Man, ‘Guidelines for managing bias in
project risk management’, in 2003 Proceedings International Symposium on
Empirical Software Engineering, 2003, pp. 272 - 280.
[24] M. Svahnberg and C. Wohlin, ‘Consensus Building when Comparing Software
Architectures’, in Product Focused Software Process Improvement, M. Oivo and
S. Komi-Sirviö, Eds. Springer Berlin Heidelberg, 2002, pp. 436-452.
[25] E. S. Andersen, K. V. Grude, and T. Haug, Goal Directed Project Management:
Effective Techniques and Strategies, 4th ed. London: Kogan Page Business Book,
2009.
[26] A. L. Jepsen and P. Eskerod, ‘Stakeholder analysis in projects: Challenges in
using current guidelines in the real world’, International Journal of Project
Management, vol. 27, no. 4, pp. 335-343, May 2009.
[27] J. Li, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse, ‘A State-of-
the-Practice Survey of Risk Management in Development with Off-the-Shelf
Software Components’, IEEE Transactions on Software Engineering, vol. 34, no.
2, pp. 271-286, 2008.
[28] B. Boehm and C. Abts, ‘COTS integration: plug and pray?’, Computer, vol. 32,
no. 1, pp. 135-138, 1999.
[29] F. D. Patterson and K. Neailey, ‘A Risk Register Database System to aid the
management of project risk’, International Journal of Project Management, vol.
20, no. 5, pp. 365-374, Jul. 2002.
[30] B. W. Boehm, ‘Software risk management: principles and practices’, IEEE
Software, vol. 8, no. 1, pp. 32-41, 1991.
[31] L. C. Ballejos and J. M. Montagna, ‘Method for stakeholder identification in
interorganizational environments’, Requirements Eng, vol. 13, no. 4, pp. 281-297,
Sep. 2008.
[32] L. W. Smith, ‘Project clarity through stakeholder analysis’, Crosstalk The
Journal of Defense Software Engineering, p. December 2000.
[33] E. Manowong and S. Ogunlanas, ‘Strategies and Tactics for Managing
Construction Stakeholders’, in Construction Stakeholder Management, Wiley-
Blackwell, 2010, pp. 121-137.
[34] I. Sommerville, ‘Models for Responsibility Assignment’, in Responsibility and
Dependable Systems, G. Dewsbury and J. Dobson, Eds. London: Springer
London, 2007, pp. 165-186.
[35] I. Sommerville, ‘Causal Responsibility Models’, in Responsibility and
Dependable Systems, G. Dewsbury and J. Dobson, Eds. London: Springer
London, 2007, pp. 187-207.
148
[36] I. Alexander and S. Robertson, ‘Understanding project sociology by modeling
stakeholders’, IEEE Software, vol. 21, no. 1, pp. 23-27, 2004.
[37] H. Sharp, A. Finkelstein, and G. Galal, ‘Stakeholder identification in the
requirements engineering process’, in Proceedings of Tenth International
Workshop on Database and Expert Systems Applications, 1999., 1999, pp. 387-
391.
[38] J. Kontio, G. Getto, and D. Landes, ‘Experiences in improving risk management
processes using the concepts of the Riskit method’, in Proceedings of the 6th
ACM SIGSOFT international symposium on Foundations of software
engineering, Lake Buena Vista, Florida, United States, 1998, pp. 163-174.
[39] J. Li, F. Bjørnson, R. Conradi, and V. Kampenes, ‘An empirical study of
variations in COTS-based software development processes in the Norwegian IT
industry’, Empirical Software Engineering, vol. 11, no. 3, pp. 433-461, 2006.
[40] M. Morisio, C. B. Seaman, V. R. Basili, A. T. Parra, S. E. Kraft, and S. E.
Condon, ‘COTS-based software development: Processes and open issues’,
Journal of Systems and Software, vol. 61, no. 3, pp. 189-199, Apr. 2002.
[41] L. Brownsword, T. Oberndorf, and C. A. Sledge, ‘Developing new processes for
COTS-based systems’, IEEE Software, vol. 17, no. 4, pp. 48-55, 2000.
[42] M. Mosko, H. Jiang, A. Samanta, and L. Werner, ‘COTS Software Acquisition
Meta-Model’, 2000.
[43] P. Ulkuniemi and V. Seppanen, ‘Definition of a COTS software component
acquisition process framework: the case of a telecommunications company’,
2002, pp. 48-54.
[44] M. Ochs, D. Pfahl, G. Chrobok-Diening, and B. Nothhelfer-Kolb, ‘A COTS
Acquisition Process: Definition and Application Experience’, 2000, pp. 335–343.
[45] J. Baik, N. Eickelmann, and C. Abts, ‘Empirical software simulation for COTS
glue code development and integration’, in Proceedings of 2001 Computer
Software and Applications Conference, 2001, pp. 297-302.
[46] B. Boehm, D. Port, Ye Yang, and J. Bhuta, ‘Composable process elements for
developing COTS-based applications’, in Proceedings of 2003 International
Symposium on Empirical Software Engineering, 2003, 2003, pp. 8-17.
[47] T. Baker, ‘Lessons Learned Integrating COTS into Systems’, in COTS-Based
Software Systems, vol. 2255, Springer Berlin / Heidelberg, 2002, pp. 21-30.
[48] M. Morisio and M. Torchiano, ‘Definition and Classification of COTS: A
Proposal’, London, 2002, vol. Lecture Notes In Computer Science; Vol. 2255, pp.
165 - 175.
[49] D. Carney and F. Leng, ‘What do you mean by COTS? Finally, a useful answer’,
IEEE Software, vol. 17, no. 2, pp. 83-86, 2000.
[50] ISO/IEC-IEEE, ‘ISO/IEC/IEEE Standard for Systems and Software Engineering
- Software Life Cycle Processes’, IEEE STD 12207-2008, 2008. .
[51] T. Gantner and T. Häberlein, ‘GARP — The Evolution of a Software
Acquisition Process Model’, Software Quality — ECSQ 2002, 2002.
[52] G. Getto, T. Gantner, and T. Vullinghs, ‘Software Acquisition: Experiences with
Models and Methods’, 2000.
[53] K. Chaves Weber, E. E. Ramalho de Araujo, D. Scaler, E. L. Pereira de Andrade,
A. R. Cavalcanti da Rocha, and M. A. Montoni, ‘MPS Model-Based Software
Acquisition Process Improvement in Brazil’, in Proceedings of 6th International
Conference on the Quality of Information and Communications Technology,
2007, pp. 110-122.
149
[54] M. A. Montoni, A. R. Rocha, and K. C. Weber, ‘MPS.BR: a successful program
for software process improvement in Brazil’, Software Process: Improvement and
Practice, vol. 14, no. 5, pp. 289-300, 2009.
[55] M. Ochs, D. Pfahl, G. Chrobok-Diening, and B. Nothhelfer-Kolb, ‘A COTS
Acquisition Process: Definition and Application Experience’, 2000, pp. 335–343.
[56] B. W. Boehm and R. Ross, ‘Theory-W software project management principles
and examples’, IEEE Transactions on Software Engineering, vol. 15, no. 7, pp.
902-916, 1989.
[57] H. Barki, S. Rivard, and J. Talbot, ‘Toward an Assessment of Software
Development Risk’, Journal of Management Information Systems, vol. 10, no. 2,
pp. 203-225, Oct. 1993.
[58] J. Ropponen and K. Lyytinen, ‘Components of software development risk: how
to address them? A project manager survey’, IEEE Transactions onSoftware
Engineering, vol. 26, no. 2, pp. 98-112, 2000.
[59] R. C. Schmidt, K. Lyytinen, M. Keil, and P. E. Cule, ‘Identifying software
project risks: An international Delphi study’, Journal of Management Information
Systems, vol. 17, no. 4, pp. 5-36, 2001.
[60] D. S. Kusumo, L. Zhu, M. Staples, and H. Zhang, ‘A Systematic Mapping Study
on Off-The-Shelf-based Software Acquisition’, in Proceedings of ACIS 2011,
Sydney, 2011.
[61] K. P. B. Webster, K. M. de Oliveira, and N. Anquetil, ‘A risk taxonomy proposal
for software maintenance’, in Proceedings of the 21st IEEE International
Conference on Software Maintenance, 2005. ICSM’05, 2005, pp. 453 - 461.
[62] T. Aven and V. Kristensen, ‘Perspectives on risk: review and discussion of the
basis for establishing a unified and holistic approach’, Reliability Engineering &
System Safety, vol. 90, no. 1, pp. 1-14, Oct. 2005.
[63] K. Georgieva, A. Farooq, and R. R. Dumke, ‘Analysis of the Risk Assessment
Methods – A Survey’, in Software Process and Product Measurement, A. Abran,
R. Braungarten, R. R. Dumke, J. J. Cuadrado-Gallego, and J. Brunekreef, Eds.
Springer Berlin Heidelberg, 2009, pp. 76-86.
[64] B. Freimut, S. Hartkopf, P. Kaiser, J. Kontio, and W. Kobitzsch, ‘An industrial
case study of implementing software risk management’, in Proceedings of the 8th
European software engineering conference held jointly with 9th ACM SIGSOFT
international symposium on Foundations of software engineering, Vienna,
Austria, 2001, pp. 277-287.
[65] P. L. Bannerman, ‘Risk and risk management in software projects: A
reassessment’, Journal of Systems and Software, vol. 81, no. 12, pp. 2118-2133,
Dec. 2008.
[66] M. Keil, L. Li, L. Mathiassen, and G. Zheng, ‘The influence of checklists and
roles on software practitioner risk perception and decision-making’, Journal of
Systems and Software, vol. 81, no. 6, pp. 908-919, Jun. 2008.
[67] M. J. Carr and S. L. Konda, ‘Taxonomy-Based Risk Identification’, Software
Engineering Institute, no. June, pp. 1-24, 1993.
[68] R. C. Williams, J. A. Walker, and A. J. Dorofee, ‘Putting risk management into
practice’, IEEE Software, vol. 14, no. 3, pp. 75-82, 1997.
[69] B. Freimut, S. Hartkopf, P. Kaiser, J. Kontio, and W. Kobitzsch, ‘An industrial
case study of implementing software risk management’, SIGSOFT Softw. Eng.
Notes, vol. 26, no. 5, pp. 277–287, Sep. 2001.
150
[70] K.-S. Na, J. T. Simpson, X. Li, T. Singh, and K.-Y. Kim, ‘Software development
risk and project performance measurement: Evidence in Korea’, Journal of
Systems and Software, vol. 80, no. 4, pp. 596-605, Apr. 2007.
[71] T. Moynihan, ‘How experienced project managers assess risk’, IEEE Software,
vol. 14, no. 3, pp. 35 -41, Jun. 1997.
[72] P. Brereton and D. Budgen, ‘Component-based systems: a classification of
issues’, Computer, vol. 33, no. 11, pp. 54-62, 2000.
[73] P. Vitharana, ‘Risks and challenges of component-based software development’,
Communication of the ACM, vol. 46, no. 8, pp. 67-72, 2003.
[74] L. Rose, ‘Risk Management of COTS Based Systems Development’, in
Component-Based Software Quality, 2003.
[75] B. Boehm, D. Port, Y. Yang, and J. Bhuta, ‘Not All CBS Are Created Equally:
COTS-Intensive Project Types’, COTS-Based Software Systems, 2003.
[76] Y. Yang, B. Boehm, and D. Wu, ‘COCOTS risk analyzer’, in Fifth International
Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems, 2006,
2006, p. 8 pp.
[77] G. Kotonya and A. Rashid, ‘A strategy for managing risk in component-based
software development’, in Proceedings of 27th Euromicro Conference, 2001, pp.
12-21.
[78] I. Sommerville, R. Lock, T. Storer, and J. Dobson, ‘Deriving Information
Requirements from Responsibility Models’, in Advanced Information Systems
Engineering, vol. 5565, P. Eck, J. Gordijn, and R. Wieringa, Eds. Berlin,
Heidelberg: Springer Berlin Heidelberg, 2009, pp. 515-529.
[79] S. Olander and A. Landin, ‘Evaluation of stakeholder influence in the
implementation of construction projects’, International Journal of Project
Management, vol. 23, no. 4, pp. 321-328, May 2005.
[80] G. M. Winch, ‘Managing Project Stakeholders’, in The Wiley Guide to
Managing Projects, Hoboken, NJ, USA: John Wiley & Sons, Inc., 2004, pp. 321-
339.
[81] A. Blyth, ‘Using Stakeholders, Domain Knowledge, and Responsibilities to
Specify Information Systems’ Requirements’, Journal of Organizational
Computing and Electronic Commerce, vol. 9, no. 4, pp. 287-296, Dec. 1999.
[82] V. Choudhury and R. Sabherwal, ‘Portfolios of Control in Outsourced Software
Development Projects’, Information Systems Research, vol. 14, no. 3, pp. 291-
314, 2003.
[83] A. Tiwana and M. Keil, ‘Control in Internal and Outsourced Software Projects’,
Journal of Management Information Systems, vol. 26, no. 3, pp. 9-44, Dec. 2009.
[84] A. Tiwana and M. Keil, ‘Does peripheral knowledge complement control? An
empirical test in technology outsourcing alliances’, Strategic Management
Journal, vol. 28, no. 6, pp. 623–634, 2007.
[85] S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian, ‘Selecting Empirical
Methods for Software Engineering Research’, in Guide to Advanced Empirical
Software Engineering, F. Shull, J. Singer, and D. I. K. Sjøberg, Eds. London:
Springer London, 2008, pp. 285-311.
[86] R. K. Yin, Case Study Research: Design and Methods, 4th ed. Los Angeles, CA:
Sage Publications, Inc, 2008.
[87] C. B. Seaman, ‘Qualitative methods in empirical studies of software
engineering’, IEEE Transactions on Software Engineering, vol. 25, no. 4, pp.
557-572, 1999.
151
[88] J. Carver, C. B. Seaman, and R. Jeffery, ‘Using Qualitative Methods in Software
Engineering’, Los Angeles, CA, USA, 2004.
[89] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén,
Experimentation in software engineering: an introduction. Kluwer Academic
Publishers, 2000.
[90] J. W. Creswell, Research design: Qualitative, quantitative, and mixed methods
approaches, vol. 3rd. Sage Publications, 2009.
[91] D. Budgen, M. Turner, P. Brereton, and B. Kitchenham, ‘Using Mapping Studies
in Software Engineering’, 2008, pp. 195–204.
[92] B. Kitchenham and S. Charters, ‘Guidelines for performing Systematic
Literature Reviews in Software Engineering’, Keele University and Durham
University Joint Report, EBSE 2007-001, 2007.
[93] K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, ‘Systematic Mapping
Studies in Software Engineering’, University of Bari, Italy, 2008.
[94] ‘Zotero’, http://www.zotero.org/, 2011.
[95] R. Wieringa, N. Maiden, N. Mead, and C. Rolland, ‘Requirements engineering
paper classification and evaluation criteria: a proposal and a discussion’,
Requirements Eng, vol. 11, no. 1, pp. 102-107, Nov. 2005.
[96] CMMI Product Team, ‘CMMI for Acquisition, Version 1.2’, SEI, Carnegie
Mellon, CMU/SEI-2007-TR-017 ESC-TR-2007-017, Nov. 2007.
[97] S. Elgazzar, A. Kark, E. Putrycz, and M. Vigder, ‘COTS Acquisition: Getting a
Good Contract’, COTS-Based Software Systems, 2005.
[98] J. S. Seibel, T. A. Mazzuchi, and S. Sarkani, ‘Same vendor, version-to-version
upgrade decision support model for commercial off-the-shelf productivity
applications’, Systems Engineering, vol. 9, no. 4, pp. 296-312, 2006.
[99] J. Holck, M. K. Pedersen, and M. H. Larsen, ‘Open Source Software
Acquisition: Beyond the Business Case’, presented at the ECIS 2005,
Regensburg, Germany, 2005.
[100] L. Morgan and P. Finnegan, ‘Open innovation in secondary software firms: an
exploration of managers’ perceptions of open source software’, SIGMIS
Database, vol. 41, no. 1, pp. 76-95, 2010.
[101] L. Briand, S. Carrière, R. Kazman, and J. Wüst, ‘COMPARE: A Comprehensive
Framework for Architecture Evaluation’, Object-Oriented Technology:
ECOOP’98 Workshop Reader, 1998.
[102] J. Holck, M. H. Larsen, and M. K. Pedersen, ‘Managerial and Technical Barriers
to the Adoption of Open Source Software’, COTS-Based Software Systems, 2005.
[103] C. Albert and L. Brownsword, ‘Meeting the Challenges of Commercial-Off-The-
Shelf (COTS) Products: The Information Technology Solutions Evolution
Process (ITSEP)’, COTS-Based Software Systems, 2002.
[104] W. Aigner, P. Regner, T. Wiesinger, and J. Kung, ‘Supporting public software
acquisition workflows - implications for data models’, in Proceedings of 15th
International Workshop on the Database and Expert Systems Applications, 2004,
pp. 1016-1022.
[105] M. Haglind, L. Johansson, and M. Rantzer, ‘Experiences integrating
requirements engineering and business analysis. An empirical study of operations
and management system procurement projects’, in Proceedings of Third
International Conference on Requirements Engineering,1998, pp. 108-117.
[106] A. Heiskanen, M. Newman, and J. Similä, ‘The social dynamics of software
development’, Accounting, Management and Information Technologies, vol. 10,
no. 1, pp. 1-32, Jan. 2000.
152
[107] G. Shaffer and G. McPherson, ‘FAA COTS Risk Mitigation Guide: Practical
Methods For Effective COTS Acquisition and Life Cycle Support’, Federal
Aviation Administration, 2002.
[108] L. D. Ball, I. G. Dambolena, and H. D. Hennessey, ‘Identifying early adopters of
large software systems’, SIGMIS Database, vol. 19, no. 1, pp. 21-27, 1987.
[109] T. Helokunnas and M. Nyby, ‘Collaboration between a COTS Integrator and
Vendors’, Software Quality — ECSQ 2002, 2002.
[110] E. Engström and P. Runeson, ‘Software product line testing – A systematic
mapping study’, Information and Software Technology, vol. 53, no. 1, pp. 2-13,
Jan. 2011.
[111] EBSE RG, ‘Template for a Mapping Study Protocol’,
http://www.dur.ac.uk/ebse/resources/templates/MappingStudyTemplate.pdf, Apr-
2009. [Accessed: 05-Mar-2011].
[112] B. A. Kitchenham, D. Budgen, and O. Pearl Brereton, ‘Using mapping studies as
the basis for further research - A participant-observer case study’, Information
and Software Technology, vol. 53, no. 6, pp. 638-651, Jun. 2011.
[113] D. Greer and R. Conradi, ‘Software project initiation and planning - an empirical
study’, IET Software, vol. 3, no. 5, pp. 356-368, 2009.
[114] D. J. Reifer, V. R. Basili, B. W. Boehm, and B. Clark, ‘Eight lessons learned
during COTS-based systems maintenance’, IEEE Software, vol. 20, no. 5, pp. 94
- 96, Oct. 2003.
[115] D. Carney, S. A. Hissam, and D. Plakosh, ‘Complex COTS-based software
systems: practical steps for their maintenance’, Journal of Software Maintenance:
Research and Practice, vol. 12, no. 6, pp. 357–376, 2000.
[116] Yi Ding and N. Napier, ‘Measurement Framework for Assessing Risks in
Component-Based Software Development’, in Proceedings of the 39th Annual
Hawaii International Conference on System Sciences, 2006, vol. 9, p. 230b.
[118] N. Admodisastro and G. Kotonya, ‘Architectural Analysis Approaches: A
Component-Based System Development Perspective’, in High Confidence
Software Reuse in Large Systems, vol. 5030, H. Mei, Ed. Springer Berlin /
Heidelberg, 2008, pp. 26-38.
[119] Vu Tran and Dar-Biau Liu, ‘A risk-mitigating model for the development of
reliable and maintainable large-scale commercial-off-the-shelf integrated
software systems’, in Proceedings of Reliability and Maintainability Symposium,
1997, pp. 361-367.
[120] L. Rose, ‘Risk Management of COTS Based Systems Development’, in
Component-Based Software Quality, vol. 2693, Springer Berlin / Heidelberg,
2003, pp. 352-373.
[121] D. Port and Y. Yang, ‘Empirical Analysis of COTS Activity Effort Sequences’,
in COTS-Based Software Systems, vol. 2959, R. Kazman and D. Port, Eds.
Springer Berlin / Heidelberg, 2004, pp. 169-182.
[122] B. Boehm, D. Port, Y. Yang, and J. Bhuta, ‘Not All CBS Are Created Equally:
COTS-Intensive Project Types’, in COTS-Based Software Systems, vol. 2580, H.
Erdogmus and T. Weng, Eds. Springer Berlin / Heidelberg, 2003, pp. 36-50.
[123] J. Li, R. Conradi, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse,
‘Preliminary Results from a State-of-the-Practice Survey on Risk Management in
Off-the-Shelf Component-Based Development’, in COTS-Based Software
Systems, vol. 3412, X. Franch and D. Port, Eds. Springer Berlin / Heidelberg,
2005, pp. 278-288.
153
[124] S. Mahmood and A. Khan, ‘An industrial study on the importance of software
component documentation: A system integrator’s perspective’, Information
Processing Letters, vol. 111, no. 12, pp. 583-590, Jun. 2011.
[125] L. Osterweil, ‘Unifying Microprocess and Macroprocess Research’, in Unifying
the Software Process Spectrum, vol. 3840, M. Li, B. Boehm, and L. Osterweil,
Eds. Springer Berlin / Heidelberg, 2006, pp. 68-74.
[126] P. L. Bannerman, ‘Risk and risk management in software projects: A
reassessment’, Journal of Systems and Software, vol. 81, no. 12, pp. 2118-2133,
Dec. 2008.
[127] D. Hillson, ‘Using a Risk Breakdown Structure in project management’, Journal
of Facilities Management, vol. 2, no. 1, pp. 85-97, 2003.
[128] M. Keil, P. E. Cule, K. Lyytinen, and R. C. Schmidt, ‘A framework for
identifying software project risks’, Communication of the ACM, vol. 41, no. 11,
pp. 76-83, 1998.
[129] P. Runeson and M. Höst, ‘Guidelines for conducting and reporting case study
research in software engineering’, Empir Software Eng, vol. 14, no. 2, pp. 131-
164, Dec. 2008.
[130] L. A. Cox, ‘Limitations of Risk Assessment Using Risk Matrices’, in Risk
Analysis of Complex and Uncertain Systems, vol. 129, Boston, MA: Springer US,
2009, pp. 101-124.
[131] B. Kitchenham, L. Pickard, and S. L. Pfleeger, ‘Case studies for method and tool
evaluation’, IEEE Software, vol. 12, no. 4, pp. 52-62, 1995.
[132] J. E. Cook, L. G. Votta, and A. L. Wolf, ‘Cost-effective analysis of in-place
software processes’, IEEE Transactions on Software Engineering, vol. 24, no. 8,
pp. 650-663, 1998.
[133] W. M. Garrabrants, A. W. Ellis, L. J. Hoffman, and M. Kamel, ‘CERTS: a
comparative evaluation method for risk management methodologies and tools’, in
Proceedings of the Sixth Annual Conference on Computer Security Applications,
1990, pp. 251-257.
[134] S. Lichtenstein, ‘Factors in the selection of a risk assessment method’,
Information Management & Computer Security, vol. 4, no. 4, pp. 20-25, Oct.
1996.
[135] M. G. Mendonca and V. R. Basili, ‘Validation of an approach for improving
existing measurement frameworks’, IEEE Transactions on Software Engineering,
vol. 26, no. 6, pp. 484 -499, Jun. 2000.
[136] K. M. Eisenhardt, ‘Building Theories from Case Study Research’, The Academy
of Management Review, vol. 14, no. 4, pp. 532-550, Oct. 1989.
[137] J. Voas, ‘Maintaining component-based systems’, IEEE Software, vol. 15, no. 4,
pp. 22 -27, Aug. 1998.
[138] Gero J.S. and Fujii H., ‘A computational framework for concept formation for a
situated design agent’, Knowledge-Based Systems, vol. 13, no. 6, pp. 361-368,
2000.
[139] K. C. Lam, D. Wang, P. T. K. Lee, and Y. T. Tsang, ‘Modelling risk allocation
decision in construction contracts’, International Journal of Project Management,
vol. 25, no. 5, pp. 485-493, Jul. 2007.
[140] M. Jorgensen, ‘Practical guidelines for expert-judgment-based software effort
estimation’, IEEE Software, vol. 22, no. 3, pp. 57-63, 2005.
[141] M. Keil and E. Carmel, ‘Customer-developer links in software development’,
Communication of the ACM, vol. 38, no. 5, pp. 33-44, 1995.
154
[142] P. Brereton, ‘The software customer/supplier relationship’, Communication of
the ACM, vol. 47, no. 2, pp. 77-81, 2004.
[143] S. Sawyer, ‘Software development teams’, Communication of the ACM, vol. 47,
no. 12, pp. 95–99, Dec. 2004.
[144] P. Waterson, S. Weibelzahl, and D. Pfahl, ‘Software Process Modelling’, in
Software Process Modeling, D. S. T. Acuña and D. N. Juristo, Eds. Springer US,
2005, pp. 111-139.
[145] K. Ballurio, B. Scalzo, and L. Rose, ‘Risk Reduction in COTS Software
Selection with BASIS’, COTS-Based Software Systems, 2002.
[146] M. John, F. Maurer, and B. Tessem, ‘Human and social factors of software
engineering: workshop summary’, SIGSOFT Softw. Eng. Notes, vol. 30, no. 4, pp.
1–6, Jul. 2005.
[147] H. Sharp and H. Robinson, ‘Some social factors of software engineering: the
maverick, community and technical practices’, SIGSOFT Softw. Eng. Notes, vol.
30, no. 4, pp. 1–6, May 2005.
[148] B. Boehm, ‘Value-based software engineering: reinventing’, SIGSOFT Softw.
Eng. Notes, vol. 28, no. 2, p. 3–, Mar. 2003.
[149] P. Grünbacher, S. Köszegi, and S. Biffl, ‘Stakeholder Value Proposition
Elicitation and Reconciliation’, in Value-Based Software Engineering, S. Biffl, A.
Aurum, B. Boehm, H. Erdogmus, and P. Grünbacher, Eds. Springer Berlin
Heidelberg, 2006, pp. 133-154.
[150] B. Boehm, P. Grunbacher, and R. O. Briggs, ‘Developing groupware for
requirements negotiation: lessons learned’, IEEE Software, vol. 18, no. 3, pp. 46 -
55, May 2001.
[151] J. Fjermestad and S. R. Hiltz, ‘Group Support Systems: A Descriptive Evaluation
of Case and Field Studies’, Journal of Management Information Systems, vol. 17,
no. 3, pp. 115-159, Dec. 2000.
[152] L. C. Ballejos and J. M. Montagna, ‘Modeling stakeholders for information
systems design processes’, Requirements Eng, vol. 16, no. 4, pp. 281-296, Nov.
2011.
[153] S. L. Lim, D. Quercia, and A. Finkelstein, ‘StakeNet: using social networks to
analyse the stakeholders of large-scale software projects’, in 2010 ACM/IEEE
32nd International Conference on Software Engineering, 2010, vol. 1, pp. 295 -
304.
[154] S. L. Lim, D. Damian, and A. Finkelstein, ‘StakeSource2.0: using social
networks of stakeholders to identify and prioritise requirements’, in 2011 33rd
International Conference on Software Engineering (ICSE), 2011, pp. 1022 -1024.
[155] S. L. Lim and A. Finkelstein, ‘StakeRare: Using Social Networks and
Collaborative Filtering for Large-Scale Requirements Elicitation’, IEEE
Transactions on Software Engineering, vol. 38, no. 3, pp. 707 -735, Jun. 2012.
[156] H. Zhu, M. Zhou, and P. Seguin, ‘Supporting Software Development With
Roles’, IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems
and Humans, vol. 36, no. 6, pp. 1110 -1123, Nov. 2006.
[157] S. T. Acuña and N. Juristo, ‘Assigning people to roles in software projects’,
Software: Practice and Experience, vol. 34, no. 7, pp. 675–696, 2004.
[158] S. T. Acuna, N. Juristo, and A. M. Moreno, ‘Emphasizing human capabilities in
software development’, IEEE Software, vol. 23, no. 2, pp. 94 - 101, Apr. 2006.
[159] A. Wise, A. G. Cass, B. S. Lerner, E. K. McCall, L. J. Osterweil, and S. M. S. Jr,
‘Using Little-JIL to Coordinate Agents in Software Engineering’, in Engineering
155
of Software, P. L. Tarr and A. L. Wolf, Eds. Springer Berlin Heidelberg, 2011,
pp. 383-397.
[160] M. André, M. G. Baldoquín, and S. T. Acuña, ‘Formal model for assigning
human resources to teams in software projects’, Information and Software
Technology, vol. 53, no. 3, pp. 259-275, Mar. 2011.
[161] C. Amrit, ‘Coordination in software development: the problem of task
allocation’, SIGSOFT Softw. Eng. Notes, vol. 30, no. 4, pp. 1–7, May 2005.
156
Appendix A: Glossary
OTS-based software acquisition: software acquisition of software that itself uses OTS
software platforms or components. For OTS-based custom software, custom
development is performed to cover remaining requirements when OTS products cannot
by themselves meet the requirements.
Risk impact: loss associated with not meeting the expected outcome.
Risk perception: the judgment that people make when they asked to characterise and
analyse risks.
157
Appendix B: Mapped papers for OTS-based software acquisition
mapping study
158
Mapped papers for OTS-based software acquisition process categories
References:
164
Appendix C: Mapped papers for risks of OTS-based and custom software development and acquisition
For each risk category, there is a possibility that a paper was mapped to a risk category more than one, because the paper had more than one risk
factor. For example, in “Planning” risk category for risks of OTS-based software acquisition, paper [17] has two risk factors: underestimated cost
estimation and multi-OTS products from different vendors may result in complicated licensing arrangements.
Risk category Risks of OTS-based software development (total mapped Risks of OTS-based software acquisition (total
papers/references) mapped papers/references)
Empirical Experience Critical framework Empirical Experience Critical framework
Planning 2 2 3
[45][53] [5][36] [17][17][36]
Requirements 24 6 5 2 2 3
[12][4][54][4][54][52][54][46] [55][14][8][22] [5][36][48][57][23] [28][44] [59][22] [17][36][17]
[52][53][44][49][45][51][52][53] [14][15]
[4][54][52][53][53][53][53][53]
Design 2 4
[14][55] [2][11][13][50]
Integration and 19 3 5 4
testing [4][54][39][35][53][4][54][6] [55][20][14] [41][50][56][3][5] [17][17][17][36]
[52][53][45][4][54][4][54][4]
[54][4][54]
System lifecycle 1 3 [1][19][38]
[53]
165
Maintenance 7 1 5 3
[4][54][53][52][4][54][53] [55] [5][37][5][3][3] [17][36][17]
Project closure 1
[54]
Software 2
[32][60]
OTS product 7 5 10 2
[52][6][6][49][6][49][53] [55][16][18] [3][3][3][7][21] [17][36]
[18][10] [36][40][50][36][50]
Cost 1 1
[3] [32]
Environment 2 2 1 1
[6][6] [16][16] [28] [17]
People 2 2 1
[6][52] [55][16] [17]
Systems 3 1
Engineering [9][9][9] [28]
Vendor 6 2 1 2 3
relationships [54][4][53][4][54][53] [55][55] [36] [28][28] [36][60][60]
Project 2 1
management [28] [28] [17]
Contract 1
[28]
Legal 1
[30]
166
Mapped papers for risks of custom-based software development and acquisition
Risk category Risks of custom-based software development (total mapped Risks of custom-based software acquisition (total
papers/references) mapped papers/references)
Empirical Experience Critical framework Empirical Experience Critical framework
Planning 1 2 1
[25] [61][33] [31]
Requirements 5 1 1 1 1
[29][24][42][24][42] [58] [31] [42] [25]
Design 1 1
[42] [31]
Integration and 1
testing [26]
System lifecycle 1
[31]
Project closure 1
[31]
Software 1 1
[31] [43]
Cost 2
[27][27]
Environment 1
[26]
People 1 1 1
[42] [31] [42]
Vendor relationships 6 2 3 2 1
[24][42][24][24][26][26] [25][25] [27][27][27] [25][47] [34]
Project management 4
[26][26][26][42]
167
Contract 1 [24] 1
[27]
Legal 1 1
[27] [27]
References:
[1] U. Lindqvist and E. Jonsson, ‘A map of security risks associated with using COTS’, Computer, vol. 31, no. 6, pp. 60-66, 1998.
[2] Guijun Wang and G. Cone, ‘A method to reduce risks in building distributed enterprise systems’, in Proceedings of Fifth IEEE
International on the Enterprise Distributed Object Computing Conference, 2001, pp. 164-168.
[3] Vu Tran and Dar-Biau Liu, ‘A risk-mitigating model for the development of reliable and maintainable large-scale commercial-off-the-
shelf integrated software systems’, in Proceedings of the Reliability and Maintainability Symposium 1997, pp. 361-367.
[4] Jingyue Li, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse, ‘A State-of-the-Practice Survey of Risk Management in
Development with Off-the-Shelf Software Components’, IEEE Transactions on Software Engineering, vol. 34, no. 2, pp. 271-286, 2008.
[5] G. Kotonya and A. Rashid, ‘A strategy for managing risk in component-based software development’, in Proceedings of 27th Euromicro
Conference, 2001, pp. 12-21.
[6] J. Bhuta, S. Mallick, and S. V. Subrahmanya, ‘A Survey of Enterprise Software Development Risks in a Flat World’, in Proceedings of
First International Symposium on Empirical Software Engineering and Measurement, 2007, pp. 476-478.
[7] J. D. Smith, ‘An Alternative to Technology Readiness Levels for Non-Developmental Item (NDI) Software’, in Proceedings of the 38th
Annual Hawaii International Conference on the System Sciences, 2005, p. 315a.
[8] V. N. Tran and D.-B. Lin, ‘Application of CBSE to projects with evolving requirements-a lesson-learned’, in Proceedings of Sixth Asia
Pacific Software Engineering Conference, 1999, pp. 28-37.
[9] B. M. Horowitz and J. H. Lambert, ‘Assembling off-the-shelf components: “Learn as you Go” systems engineering’, IEEE Transactions
on Systems, Man and Cybernetics, Part A: Systems and Humans, vol. 36, no. 2, pp. 286-297, 2006.
[10] R. Fulton, ‘Assuring certifiability of outsourced software development - a DER’S perspective’, in Proceedings of 24th Digital Avionics
Systems Conference, 2005, vol. 2, p. 5 pp. Vol. 2.
[11] B. Boehm and J. Bhuta, ‘Balancing Opportunities and Risks in Component-Based Software Development’, IEEE Software, vol. 25, no. 6,
pp. 56-63, 2008.
168
[12] P. Gruenbacher, ‘Collaborative requirements negotiation with EasyWinWin’, in Proceedings of 11th International Workshop on the
Database and Expert Systems Applications, 2000, pp. 954-958.
[13] A. Egyed, N. Medvidovic, and C. Gacek, ‘Component-based perspective on software-mismatch detection and resolution’, IEEE Software,
vol. 147, no. 6, pp. 225-236, 2000.
[14] V. Tran, Dar-Biau Liu, and B. Hummel, ‘Component-based systems development: challenges and lessons learned’, in Proceedings of
Eighth IEEE International Workshop on the Software Technology and Engineering Practice, 1997. [incorporating Computer Aided
Software Engineering], 1997, pp. 452-462.
[15] J. Akkanen, A. J. Kiss, and J. K. Nurminen, ‘Evolution of a software component - experiences with a network editor component’, in
Proceedings of Sixth European Conference on the Software Maintenance and Reengineering, 2002, pp. 119-125.
[16] W. Lam and A. J. Vickers, ‘Managing the risks of component-based software engineering’, in Proceedings of Fifth International
Symposium on the Assessment of Software Tools and Technologies, 1997, pp. 123-132.
[17] Yi Ding and N. Napier, ‘Measurement Framework for Assessing Risks in Component-Based Software Development’, in Proceedings of
the 39th Annual Hawaii International Conference on System Sciences, 2006, vol. 9, p. 230b.
[18] P. Maki-Asiala and M. Matinlassi, ‘Quality Assurance of Open Source Components: Integrator Point of View’, in Proceedings of 30th
Annual International Computer Software and Applications Conference, 2006, vol. 2, pp. 189-194.
[19] R. J. Ellison and C. Woody, ‘Supply-Chain Risk Management: Incorporating Security into Software Development’, in Proceedings of
43rd Hawaii International Conference on System Sciences, 2010, pp. 1-10.
[20] S. Blom, M. Book, V. Gruhn, and R. Laue, ‘Switch or Struggle: Risk Assessment for Late Integration of COTS Components’, in
Proceedings of Second International Workshop on the Incorporating COTS Software into Software Systems: Tools and Techniques, 2007,
p. 1.
[21] L. Merola, ‘The COTS software obsolescence threat’, in Proceedings of Fifth International Conference on Commercial-off-the-Shelf
(COTS)-Based Software Systems, 2006., p. 7 pp.
[22] V. Tran, B. Hummel, Dar-Biau Liu, Thu Anh Le, and J. Doan, ‘Understanding and managing the relationship between requirement
changes and product constraints in component-based software projects’, in Proceedings of the Thirty-First Hawaii International
Conference on the System Sciences, 1998, vol. 6, pp. 132-142 vol.6.
[23] Y. Yang, Jesal Bhuta, B. Boehm, and D. N. Port, ‘Value-based processes for COTS-based applications’, IEEE Software, vol. 22, no. 4, pp.
54-62, 2005.
[24] Jiangping Wan, Dejie Li, and K. Kuang, ‘Analysis of the Business Risks for the Software Outsourcing between Hongkong and
Guangdong’, in Proceedings of 4th International Conference on Wireless Communications, Networking and Mobile Computing, 2008, pp.
1-4.
169
[25] C. M. Lott, ‘Breathing new life into the waterfall model’, IEEE Software, vol. 14, no. 5, pp. 103-105, 1997.
[26] T. Moynihan, ‘How experienced project managers assess risk’, IEEE Software, vol. 14, no. 3, pp. 35-41, 1997.
[27] B. A. Aubert, S. Dussault, M. Patry, and S. Rivard, ‘Managing the risk of IT outsourcing’, in Proceedings of the 32nd Annual Hawaii
International Conference on System Sciences, 1999, vol. Track7, p. 10 pp.
[28] L. M. Abdullah and J. M. Verner, ‘Outsourced strategic IT systems development risk’, in Proceedings of Third International Conference
on the Research Challenges in Information Science, 2009, pp. 275-286.
[29] S. Serich, ‘Prototype stopping rules in software development projects’, IEEE Transactions on Engineering Management, vol. 52, no. 4,
pp. 478-485, 2005.
[30] T. R. Huber, ‘Reducing business and legal risks in software reuse libraries’, in Proceedings of Third International Conference on
Software Reuse: Advances in Software Reusability, 1994, pp. 110-117.
[31] Xiangnan Lu and Yali Ge, ‘Risk analysis in project of software development’, in Proceedings of the Engineering Management
Conference, Managing Technologically Driven Organizations: The Human Side of Innovation and Change, 2003, pp. 72-75.
[32] J. A. McDermid, ‘COTS: the expensive solution?’, in Proceedings of IEE Colloquium on the COTS and Safety Critical Systems (Digest
No. 1997/013), 1997, pp. 1/1-1/4.
[33] M. Jorgensen, ‘How to Avoid Selecting Bids Based on Overoptimistic Cost Estimates’, IEEE Software, vol. 26, no. 3, pp. 79-84, 2009.
[34] H. M. Edwards, G. M. Mallalieu, and J. B. Thompson, ‘Some insights into the maintenance of legacy systems within small manufacturing
and distribution organisations in the UK’, in Proceedings of The Twenty-Third Annual International of Computer Software and
Applications Conference, 1999, pp. 13-20.
[35] A. A. Kvale, J. Li, and R. Conradi, ‘A case study on building COTS-based system using aspect-oriented programming’, Santa Fe, New
Mexico, 2005, pp. 1491-1498.
[36] P. Vitharana, ‘Risks and challenges of component-based software development’, Communication of the ACM, vol. 46, no. 8, pp. 67-72,
2003.
[37] A. Stuckenholz and A. Osterloh, ‘Safe component updates’, Portland, Oregon, USA, 2006, pp. 39-48.
[38] G. Brændeland and K. Stølen, ‘Using model-based security analysis in component-oriented system development’, Alexandria, Virginia,
USA, 2006, pp. 11-18.
[39] S. Mahmood and A. Khan, ‘An industrial study on the importance of software component documentation: A system integrator’s
perspective’, Information Processing Letters, vol. 111, no. 12, pp. 583-590, Jun. 2011.
[40] F. Redmill, ‘Analysis of the COTS debate’, Safety Science, vol. 42, no. 5, pp. 355-367, Jun. 2004.
[41] M. Hepner, R. Gamble, M. Kelkar, L. Davis, and D. Flagg, ‘Patterns of conflict among software components’, Journal of Systems and
Software, vol. 79, no. 4, pp. 537-551, Apr. 2006.
170
[42] C. Schmidt, P. Dart, L. Johnston, L. Sterling, and P. Thorne, ‘Disincentives for communicating risk: a risk paradox’, Information and
Software Technology, vol. 41, no. 7, pp. 403-411, May 1999.
[43] S. A. Sherer, ‘Purchasing software systems : Managing the risk’, Information & Management, vol. 24, no. 5, pp. 257-266, 1993.
[44] J. Miller and H. C. Yeoh, ‘COTS acquisition process: incorporating business factors into COTS vendor evaluation taxonomies’, Software
Process Improvement Practice, vol. 11, no. 6, pp. 601-626, 2006.
[45] Y. Yang and B. Boehm, ‘Improving process decisions in COTS-based development via risk-based prioritization’, Software Process
Improvement Practice, vol. 12, no. 5, pp. 449-460, 2007.
[46] A. Mohamed, G. Ruhe, and A. Eberlein, ‘Optimized mismatch resolution for COTS selection’, Software Process Improvement Practice.,
vol. 13, no. 2, pp. 157-169, 2008.
[47] H. Saiedian, ‘Practical recommendations to minimize software capability evaluation risks’, Software Process Improvement Practice vol.
8, no. 3, pp. 145-156, 2003.
[48] G. Brændeland and K. Stølen, ‘A Semantic Paradigm for Component-Based Specification Integrating a Notion of Security Risk’, in
Formal Aspects in Security and Trust, vol. 4691, T. Dimitrakos, F. Martinelli, P. Ryan, and S. Schneider, Eds. Springer Berlin /
Heidelberg, 2007, pp. 31-46.
[49] J. Li, F. Bjørnson, R. Conradi, and V. Kampenes, ‘An empirical study of variations in COTS-based software development processes in the
Norwegian IT industry’, Empirical Software Engineering, vol. 11, pp. 433-461, 2006.
[50] N. Admodisastro and G. Kotonya, ‘Architectural Analysis Approaches: A Component-Based System Development Perspective’, in High
Confidence Software Reuse in Large Systems, vol. 5030, H. Mei, Ed. Springer Berlin / Heidelberg, 2008, pp. 26-38.
[51] D. Port and S. Chen, ‘Assessing COTS Assessment: How Much Is Enough?’, in COTS-Based Software Systems, vol. 2959, R. Kazman
and D. Port, Eds. Springer Berlin / Heidelberg, 2004, pp. 183-198.
[52] D. Port and Y. Yang, ‘Empirical Analysis of COTS Activity Effort Sequences’, in COTS-Based Software Systems, vol. 2959, R. Kazman
and D. Port, Eds. Springer Berlin / Heidelberg, 2004, pp. 169-182.
[53] B. Boehm, D. Port, Y. Yang, and J. Bhuta, ‘Not All CBS Are Created Equally: COTS-Intensive Project Types’, in COTS-Based Software
Systems, vol. 2580, H. Erdogmus and T. Weng, Eds. Springer Berlin / Heidelberg, 2003, pp. 36-50.
[54] J. Li, R. Conradi, O. P. N. Slyngstad, M. Torchiano, M. Morisio, and C. Bunse, ‘Preliminary Results from a State-of-the-Practice Survey
on Risk Management in Off-the-Shelf Component-Based Development’, in COTS-Based Software Systems, vol. 3412, X. Franch and D.
Port, Eds. Springer Berlin / Heidelberg, 2005, pp. 278-288.
[55] L. Rose, ‘Risk Management of COTS Based Systems Development’, in Component-Based Software Quality, vol. 2693, Springer Berlin /
Heidelberg, 2003, pp. 352-373.
171
[56] K. Ballurio, B. Scalzo, and L. Rose, ‘Risk Reduction in COTS Software Selection with BASIS’, in COTS-Based Software Systems, vol.
2255, J. Dean and A. Gravel, Eds. Springer Berlin / Heidelberg, 2002, pp. 31-43.
[57] D. Wu and Y. Yang, ‘Towards an Approach for Security Risk Analysis in COTS Based Development’, in Software Process Change, vol.
3966, Q. Wang, D. Pfahl, D. Raffo, and P. Wernick, Eds. Springer Berlin / Heidelberg, 2006, pp. 124-131.
[58] M. Feather and S. Cornford, ‘Quantitative risk-based requirements reasoning’, Requirements Engineering, vol. 8, pp. 248-265, 2003.
[59] S. Elgazzar, A. Kark, E. Putrycz, and M. Vigder, ‘COTS Acquisition: Getting a Good Contract’, in COTS-Based Software Systems, vol.
3412, X. Franch and D. Port, Eds. Springer Berlin / Heidelberg, 2005, pp. 43-53.
[60] M. A. Serva, S. A. Sherer, and J. C. Sipior, ‘“When Do You ASP?” The Software Life Cycle Control Model’, Information Systems
Frontiers, vol. 5, pp. 219-232, 2003.
[61] E. Rosendahl and T. Vullinghs, ‘Performing Initial Risk Assessments in Software Acquisition Projects’, in Software Quality — ECSQ
2002, vol. 2349, J. Kontio and R. Conradi, Eds. Springer Berlin / Heidelberg, 2006, pp. 146-155.
172
Appendix D: The framework inputs
The participant of case 1Dev is the developer (the acquirer is the stakeholder of the participant), the participant of case 2AcqBank is the acquirer
(the developer is the stakeholder of the participant), the participant of case 3AcqTelco is the acquirer (the developer is the stakeholder of the
participant), the participant of case 4AcqGovt is the acquirer (the developer is the stakeholder of the participant).
Input of the framework Case 1Dev Case 2AcqBank Case 3AcqTelco Case 4AcqGovt
Risk Developer Acquirer Developer Acquirer Developer Acquirer Developer Acquirer
Item RI RC RI RC RI RC RI RC RI RC RI RC RI RC RI RC
Selection effort ill-estimated Hi Hi Lo Lo Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi
Lo
Not adaptable to requirement
Hi (was Hi Hi Lo Lo Hi Hi Hi Hi Lo Lo Hi Hi Hi Hi
changes
Hi)
Hi
Requirements not negotiable Hi Hi Hi Hi Lo Lo Hi Hi Hi Hi Hi Hi Hi Hi Lo (was
Lo)
Complicated multi-OTS
Hi Lo Hi Hi Hi Lo Hi Lo Lo Lo Hi Hi Lo Hi Lo Lo
components arrangement
Hi Hi
Insufficient OTS component
Hi (was Hi Hi Hi Hi Hi Lo Hi Hi Lo Lo Hi Hi Lo (was
documents
Lo) Lo)
Lack of OTS-driven
requirements engineering Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Lo Hi Lo Hi
process
Maintenance planning Hi Lo Lo
Lo Hi Lo Lo Lo Lo Lo Hi Hi Lo Hi Lo Hi
unfeasible (was (was (was
173
Lo) Hi) Hi)
Hi
Upgrade unfeasible Lo Lo Lo Hi Lo Lo Lo Lo Lo Lo Lo Lo Hi Hi Lo (was
Lo)
Lack of information on provider Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Hi Lo Lo
Lo
Lack of support Hi (was Lo Hi Hi Hi Lo Lo Hi Hi Hi Hi Hi Hi Hi Hi
Hi)
Hi
Reduced control of future
Hi Lo Hi Hi Lo Lo Lo Lo Hi Hi Hi Hi Lo Hi (was Lo
evolution of the system
Lo)
Legend: RI: risk impact, RC: risk control, Hi: high, Lo: low.
174
Appendix E: Cross-case analysis for data collection 1 and 2, excluded inputs to the framework and evaluation
criteria for risk assessment
175
Q2.2.2 and Using past project data Documented in the project Integrated with budget Risk identification:
Q2.2.3 contract (not including management brainstorming and
Risk identification: brainstorming detailed technical aspects) discussion
and discussion Based on cost/revenue
Risk identification: analysis Risk analysis and
Risk analysis and prioritization: brainstorming and discussion prioritization: follow-up
discussion and using past project Risk identification: based on the discussion
data to rank risk Risk analysis and brainstorming and
prioritization: planning and discussion
review meeting
Risk analysis: what-if
scenario
178