1

Understanding Software Quality Assurance In

OSS













Behutiye Woubshet Nema
Adeyemi Bolaji
Shikur Henok
2013


2

3
Table of Contents
Abstract .................................................................................................................................... 4
1. Introduction ............................................................................................................................ 5
2. Overview of Software quality assurance ................................................................................ 7
3. Quality Assurance in Open Source Software .......................................................................... 9
3.1 Quality assurance involvements in OSS ..........................................................................12
3.1.1 Developer perspective ..............................................................................................12
3.1.2 User perspective .......................................................................................................12
3.1.3 Organization perspective .........................................................................................13
4.0 Discussion...........................................................................................................................14
5. Conclusion ............................................................................................................................17
References ...............................................................................................................................18




4


Abstract

There are lots of challenges confronting the open source software development, and one of the
problems is quality assurance. And with regards to this fact, the paper investigates the
responsible actors in assuring quality in open source software development. We meticulously
explore, evaluate and interpret the existing literature as regards quality in Open Source Software
(OSS) context.

The principal findings of the study show that sustainable community participation, development
processes, governance, systems and tools used to support development affect the quality
assurance of open source software. The paper reflects the activities which principal stakeholders
of OSS community should execute in order to achieve quality. The need to investigate on how to
encourage developers adherence towards strict development processes in OSS was also
proposed as future work. The rationale behind this is that, it will help address the discrepancy in
development process which is a key source of quality issues in OSS. The study shows the key
stakeholders that are responsible in ensuring quality in different phases of software development
in OSS and the respective activities.

5


1. Introduction

Open source communities have been rapidly attracting attentions across different fields. The
underlying reason for this unprecedented development is still not completely known, although,
many Researchers concluded that, the reason is due to low cost of software development, others
(De, 2009; Nakakoji, Yamamoto, Nishinaka, Kishida, & Ye, 2002) argued that, because it is free,
so it attracts lot of users. However, there are lots of challenges confronting Open Source
Software(OSS) projects, in general. According to (A. Khanjani & Sulaiman, 2011a), identified
one of the challenges to be in the area of quality of the software, and they also mentioned that,
one of the biggest task of OSS is assuring quality, same conclusion is reflected in literature, for
example (Nakakoji et al., 2002). Subsequently, OSS is yet to attain its full potential, as lot of
critical projects do not consider using OSS mainly because there is no strict process in assuring
quality, in other words, it seems no one is responsible for assuring quality. Further, Hedberg et
al. (2007) predicted that, with the increase in commercial use of OSS, there will more people
involved in OSS communities, particularly, the end-users with cravings for quality products
(Hedberg, Iivari, Rajanen, & Harjumaa, 2007).

According to ISO 8402 quality assurance is define as All those planned and systematic actions
necessary to provide adequate confidence that a product or services will satisfy given
requirement of quality (ISO 8402). With regards to the aforementioned definition, it is logical to
say that before quality can be attained there must be a strict process to be followed, and in the
context of software engineering this process is called software testing process. Regrettably,
Gacek and Arief in (2004) stated that OSS developers are less interested in both documentation
and testing (Gacek & Arief, 2004) processes vital in assuring quality. Herberg et al. (2007) also
supported this claim using these words usually there is no testing plan, the test coverage is not
addressed (Hedberg et al., 2007). Nonetheless, (Opensource, 2013) mentioned that OSS model
is peculiar both in organizational structure and the development process. The OSS development
process is not meticulously planned at the beginning of the project, it evolves in response to its
users demands, as contrast to the proprietary software model. OSS software development
involves distributed contributions of codes (A. Khanjani & Sulaiman, 2011a; Nakakoji et al.,
2002). Thus, the software product of OSS is highly prone to defects or bugs, because the
developers (contributors) may be following different development processes (modules).
Although, some Researchers over the years has concluded that open source development has a
greater chance of increasing the quality of product or reducing the bug, because the underlying
model in open source is that, the more people that have access to the codes, the more chances of
detecting bugs, contrast to the traditional closed source.

In addition, OSS is unique in a distinctive way, because of its structure- developer, co-developer
(who are regarded as contributor) and the end-users, these are the stakeholders in any OSS
project and they cluster around an OSS project as a community. However, without the presence
of the aforementioned principal actors, there is high possibility that a given OSS project may fail
(Nakakoji et al., 2002). In order to maintain a well-focused literature review, we have drafted
6
research questions as a guide to our review. Thus, the current study aim to investigate assuring
quality in OSS by examining different stakeholders(participant) in the development stage, with a
very focused research question of- who is responsible for quality assurance in OSS projects?.
We believe that, answering this question will help uncover the areas that need improvement in
terms of strictly following development processes (developers) or encouragement- for example
increase users participation.

We intend to conduct this study, solely by analyzing literatures with the anticipation of bringing
to light the gaps in literature as well as identifying, whose role it is to assure quality in the
different phases of software life cycle in OSS. In addition, we assumed that, the result of the
current literature review will give a broader and concise definition over who is responsible for
software quality assurance in open source software development. As result, inform both research
communities and practitioners.

The primary search process was done using online databases (the ones Oulu University
subscribed to), search engines, electronic journals, and conference proceedings. The common
online resources includes; Google scholars, Web of science, Scopus, ProQuest Databases, ACM
Digital library, IEEE xplore, SpringerLink, EBSCO, ScienceDirect, etc.
Further, papers that discussed software quality, software defect management, and software
quality assurance are the studies taken as the primary studies for the review. Additionally, the
study classified the papers based on the year of publication, the research data, research method,
and most importantly the findings. Furthermore, papers or scientific studies that are not free or in
the database research support of university of Oulu were excluded, regardless the year of
publication or the content of the paper. The papers with the above selection criteria were written
in English language, this is identified as one of the potential setbacks of the current study.

The rest of this paper is organized as follows. Section 2 provides an overview of quality
assurance in software engineering in general, as well as from open source software community
perspective. Section 3 explains in detail the factors affecting quality in OSS as describe in the
literature. Section 4 presents the discussion of the study. And finally, Section 5 presents the
conclusion of the study.

7

2. Overview of Software quality assurance

Many studies have been conducted in the context of assuring software quality (Gacek & Arief,
2004; Hedberg et al., 2007). Amidst these studies are definition of QA in software engineering,
the best practices in assuring quality, and the factors that directly impact quality- development
process, process conformance, and improvement of processes as well as validating processes.

Quality assurance is a way to assess whether the product has reached or satisfied customer
expectation. In quality assurance process there are steps that should be followed such as: design,
development or evaluation of the product by testing the result against the requirements.
Although, these processes might change depending on the project and specific companies
requirement. The whole theme of quality assurance is to provide good quality products and
services for customers. (A. Khanjani & Sulaiman, 2011b)(A. Khanjani & Sulaiman, 2011a)

Furthermore, quality assurance is a substantial part of software development as it can be related
to correctness, usability, completeness and maintainability of software which affect economic
and technical decisions made during software development. Software quality impacts a number
of important factors in our daily lives such as economy, personal and national security, health
and safety (Naik & Tripathy, 2011). Moreover, advancements in technologies and complexity of
software systems also make it necessary to understand and achieve quality assurance in our
products and development processes. This is also supported by (Otte, Moreton, & Knoell,
2008a), The execution of SQA activities requires the introduction of a quality model to obtain
process, transparency, customer satisfaction, repeatable processes and methods.

In proprietary software development process, quality assurance process includes different steps
performed by developers and testers along the development process. During the process different
standards have been applied to assure compliances with set of requirements. Performing quality
assurance has been a problem in open source software development. The method used in
traditional software development process is not pertinent to OSS development since OSS
development life cycle is unstructured and disordered. (Ardagna, Banzi, Damiani, & Frati, 2009).
Moreover, Software QA needs an integral collaboration of different elements (stakeholders) in
the software development ecosystem and a follow up of systematic procedures to attain quality.
It is highly important to ensure that processes and tools are utilized in efficient manner to achieve
quality.
Tuteja and Dubey (2012), argue that quality cannot be improved by assessing an already
completed product and highlight the importance of quality assurance measures targeted to
improve software development procedures, standards and tools used in the development. In
addition, arguing software testing techniques as measurements of software quality the authors
depict that applying software testing methods in different lifecycle stages of software
development contribute to ensuring software quality. (Tuteja & Dubey, 2012)

Further, to achieve quality product, the conformance to the development process must be strict
and followed also in distributed teams (Ardagna et al., 2009; Tuteja & Dubey, 2012). Sivert
8
(1997) defines process conformance in the context of software development, informally as the
degree of agreement between the software development process that is really carried out and the
process that is believed to be carried out (Srumgrd, 1997). The study aimed to uncover the
importance of process conformance in software engineering field. As a result the study helped to
propose four key components to measure process conformance in software development. The
components are -model for definition, measurement based on the model, an alternative
measurement and guidelines for modification of a process, so as to enable conformance
measurement (Srumgrd, 1997). Rigidly, conforming to the processes of development,
subsequently assuring quality of the software product. In addition, we can infer that, if software
products have low quality- the reason maybe that the process of development is not conformed to
or the process of development requires some improvement. Bhandari et al. (1993) selected a
narrowed aspect of software process improvement, which is defect-based oriented. They were
motivated by the inconsistency in implementation of processes in the context of software
development. However, the study responded to a practical question by a developer; I know what
I must produce; I know the process I must follow; and I have started development. How do I
detect if things are going wrong, and what do I do to get back on track? Bhandari et al., (1993),
also to create process improvement metric during development, further, the research method is a
case study of a developing team of 25 people and the main conclusion of the study was that
process improvement works (Bhandari et al., 1993). This implies that during development
process, developing team (developers) could continually improve their processes, thus assuring
quality.

In terms of development processes, it is highly important to validate the process. A relevant
study in this regard was conducted by Cook and Jonathan (1999), the study provides deeper
insight in the relationship between a formal process model and process execution. As taken from
the study; When process models and process executions diverge, something significant is
happening. They investigated amidst the metrics involved in the processes, the human or
behavioral aspects. Further, they proposed three metrics for process validation; Recognition
Metric (REC), Simple String Distance Metric (SSD) and Non-linear String Distance Metric.
These metrics were demonstrated using the Test Unit task from (Kellner et al., 1990). This is
unique in the sense that it uses the data in process validation activity (Cook & Wolf, 1994).

In addition, a recent study by Dimech et al. (2013) aimed on improving process conformance as
regards to the requirements in the context of design-driven development. They support the idea
that initial requirement can be traced from high-level design forms through the final product.
Furthermore, the authors proposed an insight of handling issues relating the problem space
among which is conformance using DiaSuite methodology (Dimech & Balasubramaniam, 2013)

9


3. Quality Assurance in Open Source Software

Unlike proprietary software development model which bases on systematic and procedural
testing, the open source software development highly utilizes post-delivery bug reports in order
to establish Quality assurance of software. Developing open source software products requires
specific methods and techniques for quality assurance process to control the quality of the
product (A. Khanjani & Sulaiman, 2011a). In addition, quick evolution of the software
development process which is mainly driven by various actors including developers, users, firms
and other participants is also another unique characteristic of the OSS model contributing to
Quality assurance. As stated by Bonaccorsi and Rossi (2003), increased participation brings
project momentum and frequent releases lead to bugs correction and generate feedback flows
that improve the code and the understanding of the underlying problem, a practice in contrast to
the conventional way of assuring quality in proprietary software (Bonaccorsi & Rossi, 2003).

A survey research on applied quality assurance methods of OSS (Otte, Moreton, & Knoell,
2008b) found out that most successful OSS projects apply structured and organized development
processes. The study also showed that engaging experienced project management, skilled
developers and users in the OSS community is substantial for quality assurance. Moreover, it
was also shown that it is essential to implement guidelines, documentations, standards and
processes to improve quality. The findings support the indispensable inclusion of a wide and
experienced OSS community together with adoption of systematic tools and development
processes to achieve software quality assurance.(Otte, Moreton, & Knoell, 2008b)

Further, high Quality Open source software development depends on fundamental sector of
sustainable community, code modularity, project management and test management. In order to
attain software quality, OSS practitioners should fully understand the dependency between these
sectors. Many researchers concluded that having a large sustainable community should be the
main objective in OSS. A research on OSS project by Mark Aberdour (2007) concluded that a
large contributor presence was one of the main factors for high OSS quality (Aberdour, 2007).
Also, Code modularity is still one of the main factors for OSS quality which programmers can
extend the program by working on separate modules, without the need to know the full
understanding of the system. This minimizes the risk of new bugs being introduced in other
modules. In addition, peer review and people management play a big role for OSS quality. Peer
review mainly focuses on assessing whether the contribution quality compliance with the code-
base. Moreover, people management plays a great role on establishing an effective environment
and culture; it is seen as important as system design. (Aberdour, 2007)

Furthermore, QA in open source software depends on two processes: code review and data
testing. The accessibility of open source software allows anyone to analyze and make thorough
assessment and get more bugs. Thus, having large community for knowledge sharing, paid and
unpaid contributors, expert developers for controlling and managing bugs, etc. For example,
open source software like GNU/Linux, Apache, Firefox, etc., receive more bugs due to the fact
10
that different contributors share their idea that leads to the possibility of increased bugs to be
released. (A. Khanjani & Sulaiman, 2011a)

Referring to the study of Nakakoji, Kumiyo, et al. (2002), a full understanding of the evolution
of an OSS system cannot be complete without understanding the evolution of the OSS
community and its roles in driving the evolution of the system (Nakakoji et al., 2002). Hence,
we aim to examine the role of each stakeholder as regard to quality of product in OSS project.
The perceptions covered in this study includes; developers- core and contributors, users and
organization. Also, this classification is informed by the popular onion model of OSS pattern.

Figure 1, adopted originally from the onion model, shows that sustainable community is
constructed from small number of core developers and increasing number of contributing
developers, bug reporters, users and organizations. Sustainable community is like multi layered
onion with users in the outer layer and move towards core through bug reporters and contributing
developers. A user increases their involvement on the basis of their knowledge through the
process. As the users move towards the core, user becomes bug reporters, contributing
developers, and core developers consecutively.

Stakeholder involvements in different stages of open source software development are explained
by drawing a metaphor based on the onion model as shown below. In order to achieve high
quality Open Source Software, the OSS community should organize processes from various
perspectives: organization perspective, user perspective, developers perspective and take
measures to assure quality (Aberdour, 2007).






Figure 1. A metaphor based on the onion model (Aberdour, 2007), shows sustainable community
in quality assurance from different stakeholder perspectives.
11


A number of models and methods have been introduced to ensure software quality. Some of the
recommended practices Hedberg et. al (2007), suggests organization and companies should
follow stricter quality assurance methods, since the commercial aspect of OSS is increasing
abundantly (Iivari, Hedberg, Rajanen, & Harjumaa, )(Hedberg, Iivari, Rajanen, & Harjumaa,
2007). In addition, it is stated that planning and documenting quality assurance process eases the
communication problem between the increasing developers and users in OSS. It is also suggested
that more emphasis should be given to testing and Test Driven Method (TDD) method should be
used in the development process. Others (Wahyudin, Schatten, Winkler, & Biffl, 2007), also
concluded that software testing and peer reviews help to reduce software defects hence software
quality is ensured.

User, developer and organizational participations are seen as the main stakeholders essential to
maintain QA of OSS. The user participation and feedback constitutes one of the backbones of the
open source model (Raymond, 2008) Different studies have showed that OSS projects that have
the support of firms and a large scale of user community benefit from participation of these
actors in assuring Software quality. Therefore, it is important to understand different aspects of
stakeholder involvements and how each stakeholders participation affects quality of the
software. Hence, with a focused research question- who is responsible for quality assurance in
OSS?

Sunindyo et al. (2013) study the business process in connections to OSS and proposed a
framework that could help OSS project managers to efficiently manage the business process.
Owing to the fact that, it is the responsibility of the OSS project managers to manage the
development process and delivering of product to the End-users. Although, this role could be
demanding, considering the collection and analysis of various data collected from different
stakeholders. Further, the proposed framework consists of four levels; data source, data
collection, data integration and the validation level. The result of the study shows that, Open
source Model framework (OSMF) can assist managers handle business process more efficiently,
as well as enhancing judgment on project quality. (Sunindyo & Ekaputra, 2013)

Abdou et al. (2012) studied popular OSS projects; Apache HTTP server, Mozilla Web browser,
and NetBeans IDE. Their focus was to analyse the test processes undertaken in those projects.
The results revealed that, three activities were in compliance to the Test Process Standard
ISO/IEC, although, significant dissimilarities were noticed in regards to the task associated with
the test process practices. They proposed a framework for software test process in OSS called
(OSS-TPF), which was compared with ISO/IEC Test process to reveal the areas that requires
improvement in achieving quality product from OSS software development. (Abdou, Grogono,
& Kamthan, 2012)

In Rashid et al. (2012), studied perception of quality of OSS product with the focus of
investigating the process practices and problems associated with the development process from
the perception of the developers. The result of the study revealed that, there is no coordination
among the developers, because of the distributed nature of OSS developmental model. However,
they highlighted the relevance of coordination between developers, thus, encourage developers
12
to coordinate, since it makes the release process systematic. As a result assist to obtain the
objective of software release process. (Rashid, 2012)

3.1 Quality assurance involvements in OSS

Different sectors of the open source software community play roles in quality assurance process.
Along community participation, governance, methodologies (development processes), systems
and tools contribute to ensuring quality in the open source software platform. However, after
reviewing processes and participations in this literature review only three key stakeholder
involvements will be emphasized to describe quality assurance participation and roles.

3.1.1 Developer perspective

Roberts et al., (2006) studied the motivations behind developer communities of the Open Source
software platform and showed that there is a significant interest for ensuring quality besides
regular contributions. The authors argue that developers interest in reputation in a specific
knowledge domain as a driving force for contributions of quality work. In other words,
developers intention to seek a status and reputation for specific skill can be related to achieving
Quality in OSS. (Roberts, Adviser-Hann, & Adviser-Slaughter, 2006)

Luyin and Sebastian (2002) studied OSS quality assurance techniques and perception of QA by
OSS developers. The study shows that, in addition to high level of user participation, developers
contribution is indispensable element in ensuring quality assurance under the OSS development
model. Discussing practices of developers, it was also shown that often OSS developers act as
users to validate functionality of software rather than following formal procedures of traditional
software testing. (Zhao & Elbaum, 2003)

3.1.2 User perspective

User participation in OSS software development is seen as crucial element that is needed to
acquire quality and improved product. OSS projects follow models that encourage user
involvement in order to facilitate development processes and assure quality. This is also
supported by different literatures and theories. Raymond E. on the Cathedral and the bazar
argues the importance of having users, Treating your users as co-developers is your least hassle
route to rapid code improvement and effective debugging. (Raymond, 2008)
Furthermore, the level of participation of users can also be related to the success and failure of
OSS projects. High level of user participation is seen as a common pattern and driving force of
QA behind most OSS projects.

Tobias et al. (2008) proposed a quality assurance process framework for ensuring quality in OSS.
The key factors emphasized in the design of the framework included, balanced interaction of
13
human factors, management skills and operating environments of organization and development
measures etc. These factors are shown to contribute to quality assurance of OSS products.
Moreover, the authors argue that QA is highly influenced due to user testing and debugging,
therefore projects need to concentrate on enabling tasks, such as structured organization, good
documentation and effective communication.

Users skill in a specific knowledge domain together with the scale of an open source project can
be used to determine the value of user participation in OSS software quality assurance.
(Jullien & Zimmermann, 2009) show how users contribution can be affecting the quality of
OSS by introducing the concept of dominant user skill. Emphasizing on user base perspective
,users skill level are used to show how product quality can be improved with bug reports,
patches( fixes) and new features from experienced users.

3.1.3 Organization perspective

Currently, different organizations are on the process of changing their critical systems and
applications to open source application, although, this decision is still highly restricted by the
quality of software in OSS. However, the underlying motivation in the adoption of OSS can be
justified by reduced costs and improved quality of software that exploits experiences of bigger
community. Sarrab et al. 2013, demands for improved and additional features along maintaining
product quality is also additional reason for organizational participation in Open source
software.(Sarrab & Rehman, 2013)

Organizations involved in development and usage of complex and mission critical software
applications find security pitfalls of the OSS as the main problem in integrating (adopting) OSS
into their platforms. And as a result, companies and enterprises are interested in pragmatic
solutions that enable assurance of OSS software. Hence, organizations will be involved in OSS
projects by initiating and making contributions to assure quality aspects of a project. Claudio et
al. (2009b) show how assurance plays a key part as requirements for telecom companies
adopting OSS for their critical systems. In addition, the authors state that The criticality and
complexity of Telco systems (e.g., for OSS/BSS systems) require the analysis and control of
software quality during the whole development process (ensured by the quality or assessment
levels of the factory or of the partner companies or vendors)(Ardagna et al., 2009b). This shows
organizational participation and contribution have higher importance in ensuring quality of OSS
projects.

14

4.0 Discussion

In this literature review, a research question for determining principal actors of quality assurance
behind open source software was established as a basis of the study. And hence, areas related to
quality assurance, development processes and community interest of participation have been
covered in the study to determine roles and participation in ensuring quality in OSS.

The principal findings of the study show that sustainable community participation, development
processes, governance, systems and tools used to support development affect the quality
assurance of open source software. By emphasizing on interests behind participation and
processes, three stakeholder involvements have been used to determine OSS quality assurance
roles and responsibilities. However, the findings also showed that governance models, level of
user and developer participation and scale of project affect the quality assurance of open source
software as shown in Figure 2. These findings also align with Aberdour (2007), which argue
that High-quality OSS relies on having a large, sustainable community results in rapid code
development, effective debugging, and new features (Aberdour, 2007).




15


Figure2.Factors influencing software quality in OSS development

The participation of different communities: users, developers and organizations together with
governance, systems and tools used in OSS development play an important role in assuring
quality process.

In order to support the idea of sustainable community participation behind quality assurance in
OSS, we can further analyze the relationship between different actors and activities involved in
OSS projects. As it can be shown from Table 1, it is essential to have participation of
organizations and developers in development and post -release phases. In these stages, the actors
(quality assurers) can enhance quality of software by creating coordination, utilizing efficient
systems and tools (test processes, quality code, patches.), following governance models that
encourage contributions etc. User participation (end user) is also shown in activities related to
bug reports, new feature suggestions, detecting bugs which mainly occur in post release phases
of OSS projects. Moreover, the table depicts interdependence among the (quality assurers) actors
in ensuring quality of OSS.


QA Assurer Activities Involvement
Developers
(core-developer, contributor)
Govern
Code review
Peer review
Develop test process
Strict development
process
Coordination
Bug report
Patch fixes
Development
&
Post -release
phase
Users
(end users)
Detect defect
Report bugs
Suggests features
Post release
phase
Organizations Govern
Coordination
Develop process
Test
Sponsor(Fund)

Development,
post release
Table1. Quality assurance activities in OSS and stakeholder involvements

16
Table1 above shows the recommended activities that each stakeholder should perform to ensure
software quality in OSS. The core developers and contributors have more similarities in terms of
activities recommended, however, the contributor is required to follow strictly the development
process, as well report bug. Similarly, the End- user is expected to also report bugs and suggest
additional features. Organizations should also govern, coordinate and develop processes.

As reflected in Table 1. There is a need to investigate how to encourage developers to follow
development process strictly in OSS. This will address the discrepancy in development process
which is a key source of quality issues in OSS.

Even though the study findings show that mainly organizational, user participation and developer
roles of involvement contribute to quality assurance of OSS, it lacks to support a detailed
analysis of different community participations and organizational structures of OSS models. In
addition, categorizations of the OSS community (stakeholder) involvements made in the study
concentrate on developer, user and organizational participation. However, this comes as a
drawback since there are OSS projects with defined QA teams, project management roles,
documentation roles, software testing team and support roles which are responsible to quality
assurance to different degrees.


17

5. Conclusion

In this review, we aim to answer the question who is responsible for assuring quality in OSS
project? Toward that, we meticulously explore, evaluate and interpret the existing literature as
regards quality in OSS context. Consequently, the existing knowledge revealed key factors that
have direct influence in assuring quality in OSS, these factors include: developers, contributors,
organization, and the End-users. It is equally important to emphasize that, OSS project is
different from traditional software development because it is situated within a community, and as
such a holistic approach should be employed when investigating OSS project

The major hindrance in assuring quality in OSS is the fact that there is no formal planning, along
with developers lack of conformance to development processes, consequently, making
assurance of quality uncertain. To this end, the existing literature suggested important activities
that should be performed by the aforementioned stakeholders in the preceding paragraph.

The findings are in accordance with the literatures explored in this review, that developers
should provide a development plan, as well as encourage the contributor to strictly adhere to the
process. Similarly, contributors in the other hand are advised to follow development process (if
any), when developing software, they should also report bugs, fix patches. Finally, the End-users
are expected to report defects and suggest features.

The contribution of this literature can be observed from two perspectives- first, for practitioners
this review enhances decision making process in terms of quality by identifying different
activities that should be performed by the key stakeholder, as well as it uncover areas for
improvement. Secondly, for academia, it provides deeper understanding of how quality
assurance is practiced in reality, and also it suggests future work.

Future research should focus on raising quality assurance awareness in the open source
community and also encouraging uniformity of development processes among developers.





18





References

Abdou, T., Grogono, P., & Kamthan, P. (2012). A conceptual framework for open source
software test process. Computer Software and Applications Conference Workshops
(COMPSACW), 2012 IEEE 36th Annual, 458-463.
Aberdour, M. (2007). Achieving quality in open-source software. Software, IEEE, 24(1), 58-
64.
Ardagna, C. A., Banzi, M., Damiani, E., & Frati, F. (2009). Assurance process for large
open source code bases. Computational Science and Engineering, 2009. CSE'09.
International Conference on, , 3 412-417.
Ardagna, C. A., Banzi, M., Damiani, E., Frati, F., & El Ioini, N. (2009). An assurance model
for OSS adoption in next-generation telco environments. Digital Ecosystems and
Technologies, 2009. DEST'09. 3rd IEEE International Conference on, 619-624.
Bhandari, I., Halliday, M., Tarver, E., Brown, D., Chaar, J., & Chillarege, R. (1993). A case
study of software process improvement during development. Software Engineering, IEEE
Transactions on, 19(12), 1157-1170.
Bonaccorsi, A., & Rossi, C. (2003). Why open source software can succeed. Research Policy,
32(7), 1243-1258.
Cook, J. E., & Wolf, A. L. (1994). Toward metrics for process validation. Software Process,
1994.'Applying the Software Process', Proceedings., Third International Conference on the,
33-44.
De, R. (2009). Economic impact of free and open source software: A study in india.
Dimech, C., & Balasubramaniam, D. (2013). Maintaining architectural conformance during
software development: A practical approach. Software architecture (pp. 208-223) Springer.
Gacek, C., & Arief, B. (2004). The many meanings of open source. Software, IEEE, 21(1),
34-40.
19
Hedberg, H., Iivari, N., Rajanen, M., & Harjumaa, L. (2007). Assuring quality and usability
in open soruce software development. Emerging Trends in FLOSS Research and
Development, , 2-2.
Iivari, N., Hedberg, H., Rajanen, M., & Harjumaa, L.Assuring quality and usability in open
source software development.
Jullien, N., & Zimmermann, J. (2009). Firms' contribution to open source software and the
dominant user's skill. European Management Review, 6(2), 130-139.
Kellner, M. I., Feiler, P. H., Finkelstein, A., Katayama, T., Osterweil, L. J., Penedo, M. H., &
Rombach, H. D. (1990). Software process modeling example problem. Software Process
Workshop, 1990.'Support for the Software Process'., Proceedings of the 6th International,
19-29.
Khanjani, A., & Sulaiman, R. (2011a). The process of quality assurance under open source
software development. Computers & Informatics (ISCI), 2011 IEEE Symposium on, 548-552.
Khanjani, A., & Sulaiman, R. (2011b). The process of quality assurance under open source
software development. ISCI 2011 - 2011 IEEE Symposium on Computers and Informatics,
548-552.
Naik, S., & Tripathy, P. (2011). Software testing and quality assurance: Theory and practice
John Wiley & Sons.
Nakakoji, K., Yamamoto, Y., Nishinaka, Y., Kishida, K., & Ye, Y. (2002). Evolution patterns
of open-source software systems and communities. Proceedings of the International
Workshop on Principles of Software Evolution, 76-85.
Opensource, i. (2013). Open source. Retrieved 15.10, 2013, from opensource.org
Otte, T., Moreton, R., & Knoell, H. D. (2008a). Applied quality assurance methods under the
open source development model. Computer Software and Applications, 2008. COMPSAC'08.
32nd Annual IEEE International, 1247-1252.
Otte, T., Moreton, R., & Knoell, H. D. (2008b). Development of a quality assurance
framework for the open source development model. Software Engineering Advances, 2008.
ICSEA'08. the Third International Conference on, 123-131.
Rashid, M. M. (2012). Release process on quality improvement in open source software
project management. Journal of Computer Science, 8
Roberts, J. A., Adviser-Hann, I., & Adviser-Slaughter, S. A. (2006). Open source software
participation: Developer motivations and performance.
20
Sarrab, M., & Rehman, O. M. H. (2013). Selection criteria of open source software: First
stage for adoption. International Journal of Information Processing and Management, 4(4),
51-58.
Srumgrd, S. (1997). Verification of process conformance in empirical studies of software
development. Department of Computer and Information Science, the Norwegian University
of Science and Technology,
Sunindyo, W. D., & Ekaputra, F. J. (2013). OSMF: A framework for OSS process
measurement. Information and communicatiaon technology (pp. 71-80) Springer.
Tuteja, M., & Dubey, G. (2012). A research study on importance of testing and quality
assurance in software development life cycle (SDLC) models. International Journal of Soft
Computing, 2
Wahyudin, D., Schatten, A., Winkler, D., & Biffl, S. (2007). Aspects of software quality
assurance in open source software projects: Two case studies from apache project. Software
Engineering and Advanced Applications, 2007. 33rd EUROMICRO Conference on, 229-236.
Zhao, L., & Elbaum, S. (2003). Quality assurance under the open source development model.
Journal of Systems and Software, 66(1), 65-75.