A systematic review on the relationship between user involvement and system success / چک نویس برای ترجمه

a r t i c l e i n f o a b s t r a c t
Article history:

Received 29 August 2013

Received in revised form 18 June 2014

Accepted 18 June 2014 Available online xxxx

Keywords:

User involvement

Software development

Systematic Literature Review

Context: For more than four decades it has been intuitively accepted that user involvement (UI) during system development lifecycle leads to system success. However when the researchers have evaluated the user involvement and system success (UI-SS) relationship empirically, the results were not always positive.

Objective: Our objective was to explore the UI-SS relationship by synthesizing the results of all the studies that have empirically investigated this complex phenomenon.

Method: We performed a Systematic Literature Review (SLR) following the steps provided in the guidelines of Evidence Based Software Engineering. From the resulting studies we extracted data to answer our 9 research questions related to the UI-SS relationship, identification of users, perspectives of UI, benefits, problems and challenges of UI, degree and level of UI, relevance of stages of software development lifecycle (SDLC) and the research method employed on the UI-SS relationship.

Results: Our systematic review resulted in selecting 87 empirical studies published during the period 1980–۲۰۱۲٫ Among 87 studies reviewed, 52 reported that UI positively contributes to system success, 12 suggested a negative contribution and 23 were uncertain. The UI-SS relationship is neither direct nor binary, and there are various confounding factors that play their role. The identification of users, their degree/level of involvement, stage of SDLC for UI, and choice of research method have been claimed to have impact on the UI-SS relationship. However, there is not sufficient empirical evidence available to support these claims.

Conclusion: Our results have revealed that UI does contribute positively to system success. But it is a double edged sword and if not managed carefully it may cause more problems than benefits. Based on the analysis of 87 studies, we were able to identify factors for effective management of UI alluding to the causes for inconsistency in the results of published literature.

۲۰۱۴ Elsevier B.V. All rights reserved.

Contents

  1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
  2. Background and motivation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
  3. Research questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
  4. Systematic Literature Review planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
    • Primary search strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
    • Study selection criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
    • Secondary search strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00 4.4. Quality assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00 4.5. Data extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00 4.6. Data synthesis and analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
  5. Systematic Literature Review execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00 6. Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00

Corresponding author. Tel.: +61 295141860.

E-mail addresses: Muneera.Bano@student.uts.edu.au (M. Bano), Didar.Zowghi@ uts.edu.au (D. Zowghi).

http://dx.doi.org/10.1016/j.infsof.2014.06.011 0950-5849/ 2014 Elsevier B.V. All rights reserved.

۶٫۱٫ Quality attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00 6.2. Temporal attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00

  • Research methodologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
  • Data sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
  1. Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
  2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ۰۰
    • Factors for effective management of user involvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
۸٫۱٫۱٫                                  Identifying users for involvement and participation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ۰۰
۸٫۱٫۲٫                                 Perspective for user involvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ۰۰
۸٫۱٫۳٫                                 Degree and level of user involvement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ۰۰
۸٫۱٫۴٫                                Stages of SDLC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ۰۰
۸٫۱٫۵٫                                 Types of system being developed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ۰۰
  • Causes for conflicting results in empirical studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
    • Different definitions and understanding of the concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
    • Differences in research methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
    • Differences in data collection methods and measurement instruments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
  • Comparison of our SLR results with a recent mapping study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
  1. Limitations of SLR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00 10. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00
  2. Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00 Appendix A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 00

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .                             ۰۰

 

۱٫   Introduction

Since the late 70s, it is believed that user involvement in system development ensures system success [1–۳]. The idea can be traced to organizational management research, including group problem solving, interpersonal communication and individual motivation [1]. The satisfaction and acceptance of the system by those who will ultimately use it, is considered as a critical success factor for the project [4–۶]. There have been numerous studies that have supported this concept (e.g. [1–۳,۷,۸]). Users typically have significant knowledge of the application domain, the tasks they perform, work practices, context of the system use and their behavior and preferences. This form of knowledge is often tacit in nature and thus difficult to be articulated with typical elicitation techniques. User involvement in Systems Development Life Cycle (SDLC) facilitates understanding of their work environment and can improve the quality, accuracy and completeness of their requirements

[۱,۷,۹].

Various methods and techniques have been proposed that provide solutions for effective user involvement. Agile methods (e.g. extreme programming), Joint Application Development (JAD), Effective Technical and Human Interaction with Computer based Systems (ETHICS) are examples of the well known techniques [10]. A few recent initiatives involve taking users’ feedback from web repositories for development of modern day applications, e.g. for online mobile applications [11], distributed collaborative application development environment [12], software requirements evolution [13], and in service oriented domain [41].

Upon closer analysis, various instances of disagreements have been observed between the authors of the voluminous empirical literature on the topic [1,2]. The conflicts in the results are claimed to be due to the inconsistencies in research method designs [1,2], confounding effects of usage of the terms ‘‘user involvement’’ and ‘‘user participation’’ [۲,۱۴,۱۵], and other contingency factors [16]. The major cause among all of them is considered to be the lack of common understanding of the concepts and philosophies of user involvement [1,2]. ‘‘User involvement in software development and system success’’ is an intricate and labyrinthine combination of three different concepts that need to be analyzed separately in their individual and distinctive definitions.

First, the concepts related to the term ‘‘Users’’, are not considered harmoniously in all the empirical studies [17]. Users play various types of roles in organization. The typical understanding of a user is someone who would be actually using the system and her/his work and environment in some way would be effected by the system. But defining the ‘‘user’’ for a project depends on the participatory methods and techniques adopted during the project. For example, Participatory Design (PD) community defines users as ‘‘the operational workers who are affected by the system, this does not include the manager’’, but in Joint Application Development (JAD), users are ‘‘any non IS/non technical individuals in the organization who are affected by the system, this includes managers’’ [۱۸].

Second, ‘‘Involvement’’ is used inconsistently in literature as a synonym for ‘‘participation’’ and ‘‘engagement’’. The first clear distinction between user involvement and user participation was given by Barki and Hartwick [14]. They defined user involvement as ‘‘a subjective psychological state reflecting the importance and personal relevance of a system to the user’’ and user participation ‘‘a set of behaviors or activities performed by users in the system development process’’. Therefore it is not necessary that the users who are involved in the project should also participate and perform activities. Whereas ‘‘user engagement’’ has been used synonymously in the literature as an additional term to both concepts of involvement and participation [8].

Third, ‘‘Software Development’’ is a life cycle that comprises of various phases, includes many activities and is affected by various dynamic and progressive factors such as methodologies used, application domains where software will be situated, and technological changes [2]. It is widely believed that involving users during early phases of development like requirements elicitation contributes most to accurately capturing their needs [7,9]. But it is also important to involve users in other stages of the SDLC, such as design and testing, when these requirements are transformed into technical solutions [18]. In different phases of SDLC various types and levels of participation of users are required. For example, senior management may be required to be involved throughout development, and middle management and other employees (such as Subject Matter Experts), would be required for their contribution during problem identification, requirements elicitation, design and testing [2]. Uncertainty, system and project complexity are important contributors that determine the phases of SDLC for user involvement, the required level and degree of involvement and the types of activities they carry out in each phase [19].

Fourth, defining and exactly measuring ‘‘System Success’’ is not addressed uniformly in the literature. Among the popular factors used for this measurement are users’ acceptance and satisfaction with the system [20], which are highly contextual and situational.

In this paper we present a comprehensive survey of literature based on review of 87 empirical studies selected within the period 1980–۲۰۱۲٫ These studies were identified by performing a Systematic Literature Review (SLR), following the guidelines provided by the Evidence Based Software Engineering (EBSE) [21]. Our objective for this review was to analyze the diverse literature to investigate user involvement and system success relationship from the incongruous and disparate findings. The preliminary results of the systematic review were presented in [22]. This paper presents the extended and complete results of our findings from the SLR. The major contributions of our paper are as follows:

  • Providing the first ever complete SLR on the user involvement and system success relationship using EBSE guidelines.
  • The analysis of the factors for effective management of user involvement during SDLC (Section 1).
  • The analysis of the factors that have caused conflicting results in the published empirical research on the UI-SS relationship (Section 2).
  • Identifying the gaps in some of the important areas of the empirical research literature about the UI-SS relationship (Section 11).

The paper is organized as follows; Section 2 describes Background and Motivation for SLR. Section 3 outlines the Research Questions. Section 4 describes the systematic review planning phase. Section 5 describes the execution phase of SLR. Section 6 shows the characteristics of the results. Section 7 is reporting findings for research questions. Section 8 is discussion on the findings. Section 9 points out possible limitations of the SLR and Section 10 gives the conclusion and Section 11 is outlining the possible future work.

۲٫   Background and motivation

For four decades the researchers were interested to investigate the axiomatic notion of a positive UI-SS relationship [1–۳,۷,۸]. But when the empirical studies were put together to identify themes and patterns in the results, they were found to be inconsistent and incongruous [1,2]. According to Ives and Olson’s review of 22 empirical studies for the period 1959–۱۹۸۱ [۱], only 36% showed positive impact of user involvement on system success. Cavaye extended the review of Ives and Olson for the period of 1982– ۱۹۹۲ with additional 19 studies [2], and found only 37% studies showing positive results. A more recent attempt has been made by He and King to analyze 82 empirical studies [3], for the period 1974–۲۰۰۷ and their meta-analysis have showed user involvement/participation has ‘‘statistically significant positive effect on both behavioral and productivity outcome’’. But they suggested that there are various confounding factors that play their role in these results. Hwang and Thorn performed meta-analysis of 25 empirical studies for the period of 1976–۱۹۹۶ [۸], and showed a positive UI-SS relationship. Bachore and Zhou [23] synthesized the findings from 46 studies (not all of them are empirical) published during 1977–۲۰۰۸٫ But both of the reviews lack the reliability due to missing a large number of studies in their sample. Kujala [7], has pointed out that the user involvement in requirements gathering phase has a positive effect in bringing user satisfaction which leads to system success, and has provided the review of methods and approaches in practice for user involvement in early phases of development for their benefits and challenges.

None of these previously conducted reviews were following the EBSE guidelines [21]. An SLR based on the guidelines follows a rigorous and scrupulous procedure for search and selection of the sample studies in review. It is methodical and meticulous process of collecting and collating the acceptable quality published empirical studies based on a systemic protocol to reduce bias and provide transparency to the process. The process is formally documented and hence repeatable.

In parallel to our systematic review, another study was published in December 2013 [42], which investigated the UI-SS relationship in a systematic mapping study providing metaanalysis. This study strengthens some of our results but differs from our review on various points. We will provide an overview of the similarities and differences between our SLR with this mapping study in Section 8.3. All of the previously published literature reviews on the topic (except this mapping study [42]) lack this rigor of search and selection method. Our primary interest in this review was to provide the basis for our ongoing research project on the same phenomenon and to increase our understanding for the work carried out in this field in order to design a case study for further investigation of UI-SS relationship.

۳٫   Research questions

Our systematic review was exploratory in nature. We were interested to find all the empirical papers published from 1980 to 2012 that investigated and evaluated user involvement in software and system development. During planning phase of our SLR, with the help of a pilot study we developed the following questions for data extractions:

  • Is there any relationship between user involvement and the successful software systems?
  • How are the users identified and selected for involvement or participation in software development?
  • What are the perspectives of user involvement in software development?
  • What are the benefits of user involvement in SDLC?
  • What are the problems caused by user involvement in SDLC?
  • What are the challenges that prevent effective user involvement in SDLC?
  • What should be the degree/level of user involvement in software development to achieve desired results?
  • Which of the stages of SDLC, user involvement is most effective?
  • What is the impact of research method on the results of inquiry on relationship between user involvement and system success?

۴٫   Systematic Literature Review planning

The aim of this review was to thoroughly examine the empirical literature on user involvement in software development. The study was carried out by the two authors in student and supervisor role. In case of conflicts the decision was taken by the supervisor to resolve it. EBSE guidelines propose three main phases of SLR [21], planning, execution, and reporting results. During the planning phase we developed a formal protocol for conducting SLR. The protocol contained the details of our search strategy guided by the research questions, inclusion/exclusion criteria, quality assessment criteria, data extraction strategy, and data synthesis and analysis guidelines. The protocol was pilot tested for evaluating the completeness of our search string, and correctness of inclusion/ exclusion criteria and data extraction strategy. After Pilot testing the updated version of the protocol was sent to the two external

Table 1

Search terms and their synonyms.

(۱) User (۲) Involvement (۳) Software development
User Involv* Software Development;
Customer (Involvement, involve, involved, involving, >>) Software Project;
Consumer Participat* IS;
End user (participation, participate, participating, participated, >>) Information System;
End-user Contribut* IT;
  (contribution, contribute, contributing, contributed, >>) Information Technology;

SDLC;

Product Development;

IT Adoption;

IT Diffusion;

reviewers, considered as experts in the field. Minor recommended changes from the reviewers related to the scope of SLR were incorporated into the protocol. During the execution the steps of protocol were refined. Protocol can be viewed online.[1] The following sub sections document the planning phase of our review.

۴٫۱٫ Primary search strategy

Our primary search strategy had the following steps;

  • Deriving major search terms from the research questions.
  • Conduct pilot testing on major terms in order to identify relevant terms, synonyms and alternative spellings that are used in published literature.
  • Connecting the resulting terms using Boolean operators to construct a search string.
  • Selecting a range of online databases, journal archives and conference proceedings for searching. Customizing the search string for the online databases’ interfaces, to be applied on abstracts.
  • Retrieving the citations and abstracts of the results and managing these using Endnotes.

From the research question, we identified the following three major terms to be used for our searching process: (1) User, (2) Involvement, and (3) Software Project. From the major search terms, we identified the alternative terms (Table 1) and concatenating the terms we formulated the search string.

ON ABSTRACT ((user OR customer OR consumer OR ‘‘end user’’ OR end-user) AND (involv* OR participat* OR contribut*) AND (‘‘software development’’ OR ‘‘software project’’ OR ‘‘IS’’ OR ‘‘information system’’ OR ‘‘IT’’ OR ‘‘information technology’’ OR ‘‘SDLC’’ OR ‘‘product development’’ OR ‘‘IT adoption’’ OR

‘‘IT diffusion’’))

The string was customized for different online databases according to their interface requirements while keeping the logical order consistent. For primary searches we selected a range of resources to reduce search selection bias, they included; ACM Digital Library, IEEE xplore, Science Direct, Google Scholar, Citeseerx, Springerlink, MIS Quarterly. We did not apply any limit on the year of publication for our results during the primary search process. During search process we realized that the concepts in the studies conducted prior to 1980 were too obsolete to be considered for our review. Therefore during study selection we posed the limit of starting year as 1980.

۴٫۲٫ Study selection criteria

Once the results were obtained, we applied the selection criteria to filter out the irrelevant studies. We were interested to select empirical studies that investigated the UI-SS relationship that provided answers to our research questions. Both authors carried out the process independently, and for differences the decision of second author (supervisor) was considered final. Study selection process was carried out in three steps.

Step 1: The results from the primary search strategy were initially screened on abstracts only to filter the papers from any of the following category;

Not in English language.

Totally irrelevant papers that were retrieved due to poor execution of search string by online search engines [12], especially Citeseerx and Science Direct.

Editorials,              tutorials,   panels,     poster      sessions,  prefaces   and opinions.

We also excluded all the papers that were published before 1980.

Step 2: on the resultant papers from step 1 were evaluated on abstracts only to exclude the studies that were

Not from the domain of IT/CS/SE/IS.

Not following any empirical research method.

PhD and Master Theses were also excluded because relevant publications resulting from the research covered by the theses were available and included in the review.

Step 3: Duplicate papers were discarded in Endnote prior to applying the selection filter. But for duplicate publications from one study having conference and extended journal versions, we manually checked and only journal papers or the article with more details of the study were included in the final results. We also excluded papers where UI-SS was not the exact focus of inquiry rather user involvement was considered as a single factor among many others, e.g. risk management or project success or failure. We also excluded papers that focused on end-user computing and on the user interface aspects of software development only and not on the entire process.

Though it was not in our selection criteria in protocol, but during quality assessment and data extraction phase, we came across some extremely low quality and plagiarized papers (in both cases they were relevant), we decided to exclude them.

۴٫۳٫ Secondary search strategy

In order to ensure that we do not miss any of the relevant studies, we devised secondary search strategy by performing following four steps.

Step 1: Based on the retrieved results, we scanned and reviewed all the references of included studies. All the eligible citations were applied with same inclusion/exclusion criteria described in Section 4.2.

Step 2: From results of step 1 of secondary searches, we realized that the issue of user involvement in software development was more extensively researched and published in Information Systems and Management journals rather than computer science and software engineering. Therefore we decided to extend our search space by using INFORMS online (Operation Research and Management Sciences). We performed online search on the following journals’ archives; Journal of Management Science, Journal of Information System Research, Journal of Operations Research from INFORMS. We also searched Association of Information System electronic Library (AISeL) which we identified from our secondary search results.

Step 3: Furthermore, we checked DBLP publication profiles of few authors who were highly cited for their work on user involvement. These include: E. Mumford, H. Barki, J. Hartwick, M.H. Olson, J.J. Baroudi, B. Ives, G. Torkzadeh, W.J. Doll, R. Hirschheim, Khalid El Emam, L.A. Keppleman, J.D. McKeen, and S. Kujala.

Step 4: To further ensure that we do not miss any important and relevant papers; in the final step we selected three of the published literature reviews (not systematic) which are highly cited and published in top ranked journals, one from each for the last three decades [1–۳];

  1. Ives and M.H. Olson, ‘User involvement and MIS success: a review of research’, Management Science, vol. 30, no. 5, pp. 586–۶۰۳, ۱۹۸۴٫

L.M. Cavaye, ‘User participation in system development revisited’, Information & Management, vol. 28, no. 5, pp. 311–۳۲۳, ۱۹۹۵٫

  1. He and W.R. King, ‘The role of user participation in information systems development: Implications from a meta-analysis’, Journal of Management Information Systems, vol. 25, no. 1, pp. 301–۳۳۱, ۲۰۰۸٫

We scanned all the references (from 1980 onwards) in those literature reviews and papers that were eligible for consideration were treated with the same three step selection criteria described in Section 4.2.

At the end of all four steps of secondary search strategy, duplicate papers were discarded and duplicate studies were grouped together and their journal versions were selected as they provided more details.

۴٫۴٫ Quality assessment

The quality of the selected studies was evaluated based on the research method they have adopted as well as the quality of their reported descriptions as these are recognized to be the only means of quality assessment available to us [24]. Overall, we have performed a three stage quality assessment as follows:

  1. Quality of the study – Our objective was to find the empirical studies investigating UI-SS relationship to answer our RQs. Therefore the studies that have utilized poorly described research methods had to be filtered out. We reused the quality assessment checklist developed using EBSE guidelines in our previously conducted SLR [25]. Appendix B provides the Quality Assessment Checklist that we used for evaluating the papers. The checklist evaluates the studies based on their strength of reporting the details of the empirical method design and execution. The first author (student) applied the quality checklist on the selected studies with discussion and feedback from the second author (supervisor). The quality assessment was not used for scoring or ranking but rather to filter out low quality publications at the time of data extraction. All the papers that scored more than 50% were included in our review and all the others were excluded.
  2. Quality of the publication outlet – For the purpose of evaluating the quality of the outlet where the papers where published, we utilized the ERA[2] (Excellence of Research in Australia) ranking of 2010. ERA is used for evaluation of the quality of the sources and outlets where the selected papers were published and may not necessarily indicate the quality of the paper itself. To ensure the quality of the included papers, we already have assessed them through the quality checklist as described above and provided in Appendix B.
  3. Assessment of the impact of the paper – To assess the impact of the published papers, we checked their citations through Google Scholar (this will be discussed in Section 1).

۴٫۵٫ Data extraction

Based on the guidance provided in [26], we extracted three types of data; Publication details, Context description, and

Findings.

  1. Publication details (title, authors, journal/conference information, ERA rank of conference/journal, year of publication).
  2. Context description (research method, data collection method, type of system/project, stages of SDLC for data collection, types of users involved, design of data collection method and instruments).
  3. Findings (relationship of user involvement and system success, perspective for user involvement, benefits of user involvement, problems caused by user involvement, factors that hinders user involvement, degree and level of user involvement, identification and selection for user involvement).

۴٫۶٫ Data synthesis and analysis

In our study, we did not differentiate between ‘‘user involvement’’ and ‘‘user participation’’ due to the following reasons:

  1. The included studies were not homogeneous in their makingdistinctions between the concepts of ‘‘user involvement’’ and ‘‘user participation’’ as described by Hartwick and Barki [14].
  2. We consider participation as a form of involvement. Becausethe users who participate in development activities are a subset of the users who are considered involved. Both involved and participating users then belong to the larger set of stakeholders [27], i.e. Participating Users Involved Users

Coding technique was used manually to identify the relevant text in the finally included papers while reading the entire paper. Later we transformed the codes to NVivo software and further performed thematic coding and analysis [26], to answer the research questions. In synthesis we were interested to divide the results against two criteria: the year of publication (divided in three decades), and research method utilized by the study in producing the results. The reason for analyzing year of the publication was to see the overall trends in three decades for temporal view and to compare our results with the previous reviews. The reason for choosing research method for analysis was because as it has been pointed out in [1,2], choice of research method may give rise to conflicting results.

To answer RQ1, we analyzed the findings of all the 87 studies to extract their outcome of UI-SS relationship investigation. We chose to provide a higher level picture of UI-SS relationship to answer RQ1 rather than going into the details of how the users were involved in those studies or how the system success was measured. On a higher abstraction level, there were three types of findings about UI-SS relationship:

  1. Positive: indicates that user involvement positively contributes to system success.
  2. Negative: indicates that user involvement causes issues and problems in software development to such an extent that may hinder system success.
  3. Uncertain: it cannot be determined that user involvement contributes to or hinders system success.

Based on the conclusions of the empirical inquiry of the 87 studies we categorized them into the above mentioned three types (Positive, Negative, and Uncertain). We used the same categorization to analyze and present the results of RQ3 (perspectives of user involvement), RQ8 (impact of SDLC where users are involved) and RQ9 (impact of research method).

For RQ2, we checked the whole study to find out if they were investigating the identification or selection of users who would be involved or will participate in the project within their empirical inquiry.

We developed the categories for the perspective of user involvement to answer RQ3, based on their objective of involving users. To address this question, our initial focus was primarily on understanding the goals and objectives for involving users in system development. But it soon became clear that focusing on goals and objectives in isolation does not provide a coherent and complete view and that we must also take into account the needs and perceived benefits of user involvement. While we were analysing the literature for these factors we also noticed that to achieve the goals of user involvement there are many problems and challenges that need to be investigated in tandem. Hence, we concluded that RQ 3, 4, 5 and 6 are indeed interrelated and we thus present our analysis for these questions in that order.

We began by reviewing the papers to answer the question: ‘‘why involve users?’’ The most obvious and widespread answer to this question is ‘‘to achieve system success’’. But it was apparent that everyone has different view of what system success really means. There were many views and positions found in the literature on this topic. So, we began by extracting the list of benefits because most authors were using benefits as criteria for determining system success. Creating a flat list of all the extracted benefits of user involvement revealed that these benefits are perceived from different points of view. For example, some researchers were only interested in the psychological stance of this benefit as perceived by the users who are involved while others were focusing on managerial or methodological aspects. This led us to conclude that there are multiple views of benefits, problems, and challenges of the UI-SS relationship. We thus decided to build a classification of these views and attitudes to use in our analysis of questions 3–۶ and we refer to them as ‘‘perspectives’’ in this paper. These perspectives will provide a typology to classify benefits, problems and challenges for user involvement. We named these perspectives with the help of thematic coding and analysis techniques [31] from the relevant coded text. These names are the most commonly used phrases from the literature that describe the relevant positions or views of user involvement.

We extracted the list of benefits, problems and challenges of user involvement to answer RQ 4, 5, 6 that were investigated by the included studies. We analyzed them according to the categories of perspectives that we had previously developed in RQ3. It is to be noted that the perspectives were developed based on the objectives of the studies that explicitly mentioned them, whereas categorization of the benefits, problems and challenges against these perspectives is our personal effort.

To answer RQ7, we read the studies to find out the level or degree of user involvement. The literature has used both terms ‘level’ and ‘degree’ as synonyms while trying to determine the time or efforts spent by the users during the system development.

For RQ8, we extracted the information from studies which mentioned explicitly the stages of SDLC which was the focus of their empirical investigation of UI-SS relationship. For RQ9, we categorized the studies based on their research method design and then analyzed their findings for RQ1, i.e. the outcome of the relationship of user involvement on system success.

۵٫   Systematic Literature Review execution

By executing the search string on selected resources, we retrieved a total of 2776 papers in our results for primary searches. The papers that were totally irrelevant were filtered after step 1 of study selection process (Section 4.3). We were then left with 290 relevant papers. After the checks from step 2 of inclusion/exclusion criteria 69 studies remained. After screening the papers from step 3, a total of 44 studies were left from primary search results. Out of 44 relevant papers, 5 were excluded based on their low quality when evaluated against our quality assessment checklist (Section 4.4 and Appendix B) and 2 were found to be plagiarizing the work of two other papers already included. So we were finally left with 37 studies. (Appendix A, S1–S37).

We then performed step 1–۳ of secondary search strategy to ensure the completeness of our results (Section 4.3). We retrieved further 21 studies that were relevant and were missing in primary search results. After this phase our total number of included papers raised to 58 studies (Appendix A, S38–S58). In step 4 of secondary search, a comparative analysis of the references from three published literature reviews, resulted in further 29 new empirical studies which were included in those literature reviews but were missing in our results (Appendix A, S59–S87). We compared our results from 1980 onwards and used the same study selection criteria that we had set for our SLR. From 58 studies only 19 were found similar whereas 44 studies included in our results were missing in those reviews. Table 2 summarizes the comparison results from step 4 of secondary searches. After this step we ended up with a total of 87 studies for our final inclusion (for study ID references see Appendix A). Fig. 1 presents the whole SLR execution process and Table 3 presents finally selected studies with IDs assigned.

۶٫   Results

In this section we describe the characteristics of our 87 included studies.

۶٫۱٫ Quality attributes

Out of 87, 39 papers are from A* ranked journals and 16 from A ranked journals/conferences, which indicates overall high quality of the results (Fig. 2).

All the studies included in our review were those that provided sufficient information about the research method and hence scored above 50% in the quality assessment checklist provided in Appendix B.

Another measure of the quality of publications that we used is the impact on relevant research community. Number of citations to a paper is considered as an indicator for good impact.[3] In our

Table 2

Summary of secondary searches step 4 (PS: primary searches, SS: secondary searches).

Ref. # of

studies

Time span covered Missing in our results Missing from their review Overlapping
[۱] ۲۲ ۱۹۵۹–۱۹۸۱ S60 NA (citations prior to 1980) PS ? S26
[۲] ۱۹ ۱۹۸۲–۱۹۹۲ S62, S65, S66, S26, S27, S28, S29, S30, S39, S46, S51, S52 PS ? S2, S5, S15, S31, S32

SS ? S40, S43, S49

[۳] ۸۲ ۱۹۷۴–۲۰۰۷ From S59 to S87 S1, S4, S7, S8, S9, S10, S11, S13, S14, S16, S17, S18, S19, S22, S27,

S28, S29, S30, S32, S33, S34, S35, S38, S39, S40, S41, S42, S43, S44, S45, S46, S48, S49, S51, S53, S56, S58

PS ? S2, S3, S5, S12, S15, S26, S31, S36, S37

SS ? S47, S50, S52, S57

  Total   ۲۹ ۴۴ ۱۹

Fig. 1. SLR execution process.

Table 3

Summary of final selection.

Included from primary searches                                                         ۳۷                  S1–S37

Included from secondary searches                                                     ۲۱                  S38–S58

Included after comparison to traditional reviews                          ۲۹                  S59–S87

Total                                                                                                           ۸۷

results 26 papers had over 100 citations and S39 and S47 had over 1000 citations (Fig. 3). Table 4 presents the references to the studies with over 500 citations.

۶٫۲٫ Temporal attributes

Out of 87 studies, 15 belong to first decade (1980–۱۹۸۹), ۳۹ belong to second (1990–۱۹۹۹) and 33 belong to third decade (2000–۲۰۱۲). Fig. 4 presents the overall summary of the resultant studies and displays their frequencies based on their publication decade, research method and ERA rank.

۶٫۳٫ Research methodologies

Our collection has 46 surveys, 20 case studies, 11 experiments, 7 field studies, one action research, one experience report and one

Fig. 2. ERA ranking of resultant studies.

grounded theory (Fig. 5). Out of 87 studies, 46 have used survey as a data collection method. The percentages of research methods utilized in the included studies are: survey 53%, case studies 23%, experiments 13%, and field studies 8%. In the first decade, 80% of the studies are using survey research method and almost all of

 

Fig. 3. Citation count for resultant studies from Google Scholar (as on 18th August 2013).

them are of very high quality as 13 out of 15 are published in A* ranked journals (Fig. 6). In the second and third decades the trend of using other research methods has increased but the ranking of the conferences/journals where the papers are published has decreased.

Table 4

Fig. 5. Research methods in resultant studies.

۶٫۴٫ Data sources

Table 5 shows the top ten conferences/journals frequencies for our resulting studies along with their ERA rank. It is important to note that 21 of the studies included (12 from MISQ and 9 from JIM), are published in the highest ranked IS outlets with highest impact factors for many years. Moreover, as Fig. 2 indicates, our entire collection of extracted papers comes from very highly ranked and cited outlets. This is a clear indication of the rigor and quality of data sources where our collection of studies come from.

 

Top 5 highly cited studies in results with above 500 citation on Google Scholar (as on 18th August 2013).

ID Complete reference Citation count
S39

S47 S2

S34

S65

J.D. Gould and C. Lewis, ‘‘Designing for usability: key principles and what designers think,’’ Communications of the ACM, vol. 28, no. 3, pp. 300–۳۱۱, ۱۹۸۵

J. Hartwick and H. Barki, ‘‘Explaining the role of user participation in information system use,’’ Management Science, pp. 440–۴۶۵, ۱۹۹۴

J.J. Baroudi, M.H. Olson, and B. Ives, ‘‘An empirical study of the impact of user involvement on system usage and information satisfaction,’’ Communications of the ACM, vol. 29, no. 3, pp. 232–۲۳۸, ۱۹۸۶

H. Barki and J. Hartwick, ‘‘Measuring user participation, user involvement, and user attitude,’’ MIS Quarterly, pp. 59–۸۲, ۱۹۹۴ Jarvenpaa, S.L., and Ives, B. Executive involvement and participation in the management of information technology. MIS Quarterly, 15, 2 (June 1991), 205–۲۲۷

۱۴۱۷

۱۱۹۱

۷۸۳

۶۹۷

۵۸۹

  Fig. 4. Summary of characteristics of included studies (decade, research method, ERA rank).  
Please cite this article in press as: M. Bano, D. Zowghi, A systematic review on the relationship between user involvement and system success, Inform.

Softw. Technol. (2014), http://dx.doi.org/10.1016/j.infsof.2014.06.011

Fig. 6. Relationship of user involvement and system succes.

Table 5

Top ten journals in results of SLR.

# Name of conference/journal # of studies ERA rank Impact factor
۱ MIS Quarterly

http://www.misq.org/about/

۱۲ A* ۴٫۶۵۹
۲ Information and Management

http://www.journals.elsevier.com/information-and-management/

۹ A* ۳٫۱۷۸
۳ Decision Sciences

http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1540-5915

۴ A* ۱٫۴۸۴
۴ Management Sciences

https://www.informs.org/Find-Research-Publications/Journals/Management-Science

۴ A* ۱٫۷۳۳
۵ Behaviour and Information Technology

http://www.tandfonline.com/toc/tbit20/current#.UgmsPGGs7dM

۳ A* ۰٫۸۵۶
۶ Omega International Journal of Management Sciences http://www.journals.elsevier.com/omega/ ۳ A* ۳٫۰۲۴
۷ Journal of Management Information Systems http://www.jmis-web.org/toppage/index.html ۲ A* ۲٫۶۶۲
۸ Information System Research

https://www.informs.org/Find-Research-Publications/Journals/Information-Systems-Research

۲ A* ۲٫۱۴۶
۹ Information and Software Technology

http://www.journals.elsevier.com/information-and-software-technology/

۲ B ۱٫۶۹۲
۱۰ Journal of Information System

http://www.journals.elsevier.com/information-systems/

۲ B ۱٫۷۶۸

 

۷٫   Findings

Based on the data extraction from our set of 87 studies and thematic coding and analysis, we now present our findings to answer the research questions.

RQ1: Is there any relationship between user involvement and the successful software systems?

Fig. 6 presents and summarizes the results for RQ1. These results are divided into three categories based on the output of their inquiry on the UI-SS relationship: Positive (+, showing the user involvement has positive influence in bringing about system success), Negative (, showing that user involvement has negative influence on system success) and uncertain (?, the results of the studies are inconclusive, neither positive nor negative). To give a more comprehensive picture of the results obtained, the frequencies are further mapped against research methodology adopted to obtain results, and the decades to which that publication belongs.

Looking at the overall results, according to Fig. 6, out of 87 papers, 59 studies show positive impact of user involvement on system success and they make 68% of the results. 7 papers are reporting negative results, whereas 21 are uncertain on the issue. From 59 papers showing positive results, 31 used survey research method, which is almost 53% of the positive results. Analyzing the detailed breakup shows that from 1990s onwards, the research is showing more positive results. But during the first period, the trend was different. This is consistent with the other literature reviews from the last three decades [1–۳] (Section 2). In our SLR, 32% of the studies show negative or uncertain results. This variation could be due to the facts that data was collected at different stages of SDLC, organisations were of different sizes, projects were developing different types of systems, could have had varying degree of complexity and the software development methodologies were varied. The reason for these differences will be discussed later (Section 8.2) in more details.

RQ2: How are the users identified and selected for involvement or participation in software development?

For effective user involvement, it is very important to identify the right people from a group of stakeholders who are to be involved or given the chance to participate. Not all the stakeholders carry equal relevance to software being developed. Not all involved users are required to participate partially or fully in software development. Therefore from the involved users, often another subset is selected whom would be given the chance to actually participate in the development process [27].

Surprisingly our SLR results did not yield many articles that discussed the identification of users for involvement with the exception of S28 and S52. However, this topic has been covered reasonably well within the Requirements Engineering (RE) literature though not empirically tested sufficiently, for example the work of Macaulay [28], Robertson [29], and Alexander [30].

S28 gives a brief description of who can be a potential user for involvement but does not provide any details on how to select them:

‘‘A user can be either a manager or a staff specialist (e.g., corporate planner, marketing researcher, or production planner). This is the person who benefits from the output provided by the DSS. Users can vary from managers with little knowledge and/or interest in computer technology to staff specialists with extensive computer training.’’

According to S52, they found that identifying appropriate users for involvement is necessary as this creates impact on the user satisfaction level:

‘‘Systems professionals and user area management should identify users who are most likely to benefit from high degrees of user participation or leadership. The results suggest that the traditional approach of involving users … where analysts obtain information from and consult with users in order to determine requirements is sufficient for developing transaction-processing systems. The traditional user involvement approach appears appropriate for lower level users as well. The results suggest that increasing the involvement of lower level users or users of transaction processing systems does not result in increased user information satisfaction.’’

RQ3: What are the perspectives for user involvement in software development?

After performing thematic analysis, five categories of perspectives emerged as represented in Fig. 7. Not all of the papers reviewed in the SLR had articulated an explicit perspective, so this figure does not include all 87 studies. The highest number of positive results in the UI-SS relationship comes from the studies that focus on psychological perspective. Another interesting point to note is that cultural perspective does not seem to yield any

positive results. We now give a brief description of these emerging perspectives. These perspectives play an important role in the analysis of, and discussion on, the benefits, problems and challenges of user involvement.

  • Psychological perspective

One of the main factors in defining system success is users’ satisfaction, that is, considering a psychological state that comes when users perceive that they have control over the system development process [14]. Therefore involvement (where users may not be participating or engaged) actually gives users’ satisfaction by their sense of control (S32). The other psychological factors that play a role in user involvement are their willingness to participate, their capability which ultimately impacts their interest, and their characteristics and attitudes [2].

  • Managerial perspective

The users are not to be merely involved, their involvement and participation has to be properly managed to achieve the desired results of bringing about system success. The focus of management is the identification of users or their representatives, selecting a method or technique for their ‘‘effective’’ involvement, and managing their participation in the required activities to ensure the achievements of objectives [5]. Managers have to take into account who should be involved in what stages of development, the allocation of financial resources needed, and to ensure higher level of management commitment and support [5].

  • Political perspective

The degree of power given to users during their project involvement will undoubtedly determine their extent and degree of influence on the outcome of the project. The level and degree of involvement can be affected by organizational or political influence especially when it comes to power of decision-making and implementing changes. There can be potential conflicts between users and the development team and it is imperative that conflict resolution strategies are available when this occurs (S43).

  • Cultural perspective

Although our findings reveal that the concept of culture has not

 

Fig. 7. Perspectives of user involvement.

 

been studied as widely and as frequently as other factors but it remains as one of the effective perspectives that needs to be addressed in user involvement. The overall purpose of user involvement can be different when linked to different cultural contexts, be it national, organisational or project cultures. Organizational culture influences the consideration of why and how to involve users (S18). The two most famous cultural based practices are Scandinavian Participatory Design with strong Marxist flavor and American Joint Application Development [7,18,32].

  • Methodological perspective

The main factor for selection of particular method for user involvement depends on the intensity of involvement required in the software development process. Other factors that play a role in this selection are varying degrees of project complexity, and available technological resources [2]. For informative role (where users only provide information related to the end product), interviews, focus groups, and surveys can be used. For participative role (where users are performing activities during development and have some power to influence decision making), agile methods, and Participatory Design techniques can be used. For

 

Table 6

Benefits of user involvement.

  Benefits of user involvement Description Extracted from following studies Freq

(N = 87)

Benefits from psychological perspective

Benefits from managerial perspective

Benefits from methodological perspective

Benefits from cultural perspective

Benefit from political perspective

User system satisfaction

User system acceptance

Facilitating change

Better user’s attitude towards system

Increasing perceived relevance to the system by users

Increasing user motivation

Increasing customer loyalty

Assist in maintaining long term relationship with users

Better communication

Improved Management

Practice

Developing realistic expectation

Reducing cost of the system

Helping in conflict resolution

Better understanding of user requirements

Improving quality of resultant application

Improving quality of design decisions

Helping in overcoming in implementation failures

Increased system usage

Facilitating knowledge sharing

Improving user’s skills

Democracy in workplace

Users will favor a system more if they are involved in its development and feel satisfied with using it

Users approve that the system is developed according to their workplace needs and requirements

Involved users will not resist using a new system in their work environment Involved users will show positive attitude when using the system

Involved users considered themselves more informed about the system and think that the system is relevant

Involved users will be more motivated to use the system

Involved users will have higher degree of trust in the development team

Involved users will have more interaction with the development team. This helps maintain long term relationships between users/ customers and development team

User involvement will lead to increase in interaction between users and development team and will facilitate more effective communication

By involving the users in the development, the management will face less resistance by giving the users sense of dignity of knowing that they are important for the system

Users will have a more informed idea of the features of the system being developed Decreasing the risk of too many changes after implementation by involving users in the project

User involvement can help resolve disagreements that may arise between users and developing teams

Eliciting more accurate requirements from the users of the systems

By involving the users the non functional aspects of the system such as functional

suitability, reliability, usability, performance, efficiency, compatibility, security, maintainability and portability can be elicited which may not have been expressed explicitly hence improving the quality of the system Based on the level of users understanding, skills and their workplace environment the decisions for the design of the system will be better informed

When users are part of the testing, implementation and installation of the system, this can reduce the number of failures

Having the sense of involvement will help in the increase system usage in the workplace environment

Learning from users about their work place practices, domain, organization and teaching them about the system being developed Training the users for system utilization

Giving users the ability to influence decisions and give them sense of empowerment so that they feel the ownership of the system and have a sense of control over the development process

S3, S13, S16, S20, S21, S27, S33, S34,

S35, S37, S38, S45, S46, S52, S59, S63,

S65, S67, S68, S71, S83, S84

S4, S11, S13, S38, S40, S43, S46, S64,

S87

S5, S12, S69, S71, S72 S5, S12, S69, S71, S72

S12

S16

S21

S21

S10, S12, S25, S55, S58, S77

S16, S29

S32, S56

S43, S52

S32

S8, S10, S14, S16, S21, S22, S37, S38,

S41, S43, S45, D46, S50, S57, S64, S70, S71, S75, S79, S83

S11, S26, S27, S36, S37, S38, S40, S52, S57, S68, S70, S71, S77, S79, S83, S87

S6, S9, S11, S40, S41, S46, S52, S64,

S65, S69, S77, S83

S31

S2, S5, S12, S26, S34, S35, S39, S46,

S47, S60, S62, S63, S77, S84

S3, S5, S8, S9, S10, S14, S41, S45, S50,

S56, S57

S40, S46, S64, S67

S6, S16, S18, S23, S56, S65, S75

۲۳

۹

۶

۵

۱

۱

۱

۱

۶

۲

۲

۲

۱

۲۰

۱۶

۱۲

۱

۱۴

۱۱

۴ ۷

consultative role (where users have to provide feedback or reviews), meetings can be arranged [33]. Muller et al. [10] has provided the taxonomy for 61 participatory practices encompassing various stages of SDLC. According to Cavaye [2], what makes us chose one method or technique over others depends on what is referred to as the underlying philosophy for user participation such as functionalist versus the neo-humanist paradigm [2].

RQ4: What are the benefits of user involvement in SDLC?

In Table 6, we present the extracted list of benefits of user involvement classified according to the perspectives. This

Please cite this article in press as: M. Bano, D. Zowghi, A systematic review on the relationship between user involvement and system success, Inform.

Softw. Technol. (2014), http://dx.doi.org/10.1016/j.infsof.2014.06.011

 

classification is based on thematic coding and analysis that we carried out to answer questions 3–۶٫

Obviously, the focus of all the studies was to achieve system success by involving users. What exactly meant by ‘‘system success’’, is highly contextual and depends largely on many factors. It has long been recognized that system success cannot be assessed and measured purely in economic terms by considering return on investment (ROI). This is partly due to the fact that it is hard to compare ROI of a system with alternative investment opportunities. As Cavaye [2], states this form of evaluation is hard because intangible costs and benefits of systems are hard to identify and difficult to articulate in financial terms. That is the main reason why ‘‘user satisfaction’’ is the most widely used alternative to measure system success in the majority of empirical studies. Our SLR confirms this fact with 23 of the empirical studies stating user satisfaction leads to system success. However, user satisfaction has also been problematic as a measure for system success for many reasons identified in the literature [34].

RQ5: What are the problems caused by user involvement in SDLC?

User involvement is a double-edged sword that if mismanaged, it can cause serious problems rather than benefits. Table 7 enlists these problems reported in our resultant studies. As this table presents, not all perspectives are covered because many of the previous empirical studies were not investigating these perspectives for problems. Also it is important to note that not all 87 studies are listed in this table. The most prominent problems caused by user involvement is communication problems and misunderstanding between the users and the development teams leading to all kinds of conflicts. This may also have an impact on the degree and level of user involvement and may determine which stage of SDLC they are likely to be involved (discussed in RQ7 and RQ8).

RQ6: What are the challenges that prevents effective user involvement in SDLC?

  Problems caused by user involvement Description Extracted from following studies Freq

(N = 87)

Psychological perspective Users’ negative attitude Users will resist the implementation of new system or change in existing system S31, S50, S62, S83 ۴
  Users’ expectations Users may pose unrealistic expectations from the system S1, S83 ۲
  Difference in perception in level of involvement Users may not feel sufficiently involved in the project leading to negative attitude towards the system S13 ۱
  Lack of cooperation Users may not cooperate with the development team S22 ۱
  Manipulation of users Development team may try to manipulate users into making them accept the system that may not satisfy their requirements S33 ۱
  Intergroup hostility Personality clashes between the users and the developers may lead to hostility in both groups S80 ۱
  Negative perception by users Users having a negative perception about the development team due to problems in communication S80 ۱
Managerial perspective Misunderstandings Problems in communicating between the users and the developers, causing misunderstanding of each other’s views S3, S7, S10, S14,

S50, S79, S80

۷
  Conflicts Disagreement of opinions and goals that may arise during user involvement S7, S21, S43, S49,

S80

۵
  Management problems Keeping users involved throughout the project will give rise to various managerial problems S14, S50 ۲
  Cost of user training Training the users about the nature of their involvement may put additional cost on the project budget S79 ۱
Political perspective User’s influence on decision Problems that would arise between the users, management and the development team due to the conflicts in the level of influence of users on the decisions S7, S43, S53, S72,

S73

۵
  Ignoring users’ feedback Ignoring users’ feedback and their requirements about the system will cause a negative impact on their attitude towards the system S12 ۱

Table 8 presents all the factors given in the studies that are considered challenging to effective user involvement. All five perspectives are represented in this table. The top challenge that hinders effective involvement of users is their lack of motivation

Table 7

Problems caused by user involvement.

followed by problems with their attitude and behavior. We are only interested in presenting these challenges as they were listed in the reviewed literature. But it would be an interesting exercise to develop a cause and effect model of all these challenges to see the relationship between them.

RQ7: What should be the degree/level of user involvement in software development to achieve desired results?

Papers that have investigated this research question have essentially referred to the works of different authors for discussing the appropriate level and degree of user involvement. The highest citations for this concept are those of Enid Mumford [35] (cited by S4, S9, S14, S18, S23, S33, S71, S72, S83, S84), Ives and Olson [1] (cited by S21, S31, S56), and Damodaran [33] (cited by S21, S23). Hence we now provide an overview of the seminal classification proposed by these three publications.

Enid Mumford is the highest cited author for her work on providing first distinction of user roles and providing further classification of direct and indirect user participation. Indirect user participation is achieved by representing users by intermediary. She has proposed three types of participation starting from lowest to indirect and highest to most direct [35]: ‘‘Consultative, where design decisions are made by the systems group, but the objectives and form of the system are influenced by the needs, especially job satisfaction needs, of the user department; Representative, where all levels and functions of the affected user groups are represented in the system design team; Consensus, where an attempt is made to involve all workers in the user department, at least through communications and consultation, throughout the system design process.’’

According to Ives and Olson, the degree of user involvement refers to ‘‘the amount of influence the user has over the final product’’, and thus would ultimately contribute to efficiency of user involvement [1]. They have divided the level/degree of user involvement into following types; ‘‘No involvement: users are unwilling or not invited to participate; Symbolic involvement: user input is requested but ignored; Involvement by advice: advice is solicited through interviews or questionnaires; Involvement by weak control: users have sign-off responsibility at each stage of the system development process; Involvement by doing: a user is a design team member, or

Table 8

Challenges of user involvement.

  Challenges for user involvement Description Extracted from following studies Freq

(N = 87)

Psychological perspective Users’ lack of motivation Users may not wish to participate or get involved in the project S3, S4, S9, S40, S63,

S64, S68, S70, S78,

S83

۱۰
  User behavior problems Users may not have the right attitude to the workplace causing behavioral problems S10, S36, S47, S62,

S73, S87

۶
  Users expertise Users may not have the right level of expertise to participate or get involved and may feel intimidated and resist to the system S63, S70 ۲
  Legacy thinking Users may not appreciate the idea of change in their existing work environment S4 ۱
  Confidentiality concerns Users may not feel comfortable sharing their views about their work practices S9 ۱
  Users expectations Users may have unrealistic expectations from the system S14 ۱
Managerial perspective Time constraints User involvement always requires more time S3, S52, S54, S59 ۴
  System complexity It is difficult to involve users in development of a large and complex systems S8, S31, S37, S70 ۴
  Communication skills of users Users may not have the right communication skills to let the development team know about their needs S9, S35, S37, S79 ۴
  Budget User involvement always incurs additional cost S4, S16, S33, S59 ۴
  Lack of top management support Top level management may not be supportive of involving the users S63, S87 ۲
  Project uncertainty Project uncertainty makes it difficult to manage effective user involvement S38 ۱
  Efforts required by users User participation requires extra work on their part which may not be possible for them S52 ۱
  Ineffective user representation Incorrect selection of the people to represent the users S54 ۱
  User training Additional training cost and efforts may be required S27 ۱
Methodological perspective Task complexity User involvement in a project where users tasks are highly complex would be challenging S37, S62, S87 ۳
  User identification Identifying the right users (who are available) for involvement and participation is a challenging task S28 ۱
Cultural perspective Impact of Change The development of new system will bring change in the work environment and it would be challenging to introduce and communicate the impacts of such changes. S31, S61, S63 ۳
Political perspective Degree of involvement There might be differences in opinions on the level and degree of involvement of users in a project and to what extent they can influence the outcomes S8, S37 ۲
  Conflicts There is always a chance that conflicts arise between users and development team and effective conflict resolution is required from management S21 ۱
  Power asymmetry In case of power asymmetry between users and developers, the involvement and participation may not yield the right results S7 ۱

is the official liaison with the information systems development group; Involvement by strong control: users may pay directly for new development out of their own budgets, or the user’s overall organizational performance evaluation depends on the outcome of the development effort.’’

Damodaran has considered user involvement as falling on some point on the continuum from informative, through consultative, to participative [33]. These levels are not restricting the type of involvement of physical presence of the user. On the contrary, the form of involvement basically describes the way in which the users are involved. The author has identified three levels of involvement [33]; ‘‘Informative: users provide and/or receive information. In other words, users are indirectly affecting the system design process, instead of physically participating in the design activities; Consultative: users comment on a predefined service or range of facilities. In the context of this article, predefined service or range of facilities are considered as any type of artifact produced or developed during the design process; Participative: users influence decisions relating to the whole system. To directly influence the decision-making process of system design, a concrete participation in the design process is assumed. Users are most likely an integrated part of the design team residing in the development organization’s facilities, but most importantly, part of the system design process.’’

Study S23, builds on the work of Clement [36], besides Mumford and Damodaran. Clement has considered political dimension of the concept of user participation and referred to it as user empowerment. He distinguishes user participation into two categories; functional and democratic. For functional empowerment, Clement states: ‘‘the users should be able to carry out their work to their own satisfaction and in an effective, efficient, and economical manner’’ and democratic empowerment is defined as ‘‘users should have the mandate to participate in decision-making in their workplace including the design and development of software and IT-based systems’’ [۳۶].

In summary, our analysis of RQ7 did not reveal any universally accepted answer to this question. By that we mean that the degree and level of user involvement fits within a spectrum (from no involvement to full involvement leading to complete participation). The answer to this question is therefore: ‘‘it depends’’. This broad spectrum is contextual in nature and depends on many factors in each individual software development project such as users’ expertise, their previous experience, and organizational culture and politics as well as task uncertainty.

RQ8: Which of the stages of SDLC, user involvement is most effective?

Not all the studies explicitly mentioned the stage(s) of SDLC that they focused for evaluating the UI-SS relationship. Fig. 8 represents the broad level categorization of the studies that explicitly mentioned the phases of SDLC i.e. Requirement Analysis (comprising of project planning/scoping and requirements elicitation, analysis and verification), Design (software design and Architecture), and Implementation (comprising of coding, testing and installation). Fig. 8 represents the relevant studies according to their outcomes of UI-SS relationship (i.e. positive, negative or uncertain).

The participation takes different forms in different stages of software development, depending on the type and level of user participation with respect to the stage of SDLC (S62). Although user involvement and participation have been recommended throughout the SDLC, but it is considered to be most effective in the early stages such as requirements analysis and design (S6, S38). The literature claims that after effective management of user involvement or participation in one stage of SDLC is said to influence the level of participation in the subsequent stages [44]. Some researchers have stressed that involving users in early stages of SDLC is more important and beneficial than others [45–۴۷], and stressed that after effective involvement of users in RE further involvement may not be required in subsequent phases [48]. But the there is not any evidence that would compare the phases of SDLC and demonstrate in which phase of SDLC user involvement would be most effective [43].

Another important aspect of user involvement is their participation in testing phase. The included studies were not solely focusing on testing phase and it was considered the part of implementation of system on a broader level. With exception of the studies mentioned in Fig. 8, the rest of them have investigated user involvement either throughout the SDLC or have not given any information in this regard. We scanned all the studies to extract the information about the types of tests they performed where users were involved. S5, S6, S22, S29, S30, S77 have included a variable to measure whether users were involved in testing (or implementation) phase or not. Various types of testing techniques have been found in empirical investigation of some of the studies i.e. Usability Testing (S6, S9, S21, S25, S39, S53, S83), User Acceptance Testing (S13, S14, S23, S26, S42), Prototype Testing (S10, S39, S42), Beta Testing for obtaining feedback (S21, S83), Test Driven Development (S54), System Testing (S32) and Cooperative Evaluation Techniques (S9).

There are also differences of opinions among various methodologies for user involvement and they recommend different stages for user involvement to be more effective. For example, JAD focuses on meeting with users in requirements stage whereas PD considers involvement throughout the entire project for visible benefits [18].

To conclude, we did not find any convincing or compelling evidence in the empirical studies that would illustrate with absolute certainty which phase(s) of SDLC user involvement is most effective.

RQ9: What is the impact of research method on the results of inquiry on relationship of user involvement and system success?

Methodological problems arise when the research design fails to compensate the weaknesses associated with a particular research method. A good research design comprises of methods suitable for a particular research problem, tapping into its

strengths, while mitigating its weaknesses. The reliability and the validity of the results of a research design depend largely on how well the researchers have tried to compensate for the weaknesses of the method [49].

Our results show that the choice of research method has had an impact on investigation of the UI-SS relationship (Fig. 6). From 59 papers showing positive results, 31 used survey research method, which is almost 53% of the positive results. Surveys are used to collect data from large sample to test a hypothesis but they are not considered appropriate for exploring relationships among complex phenomenon [7]. They only provide surface level opinions from respondents. Case studies are considered more effective for investigating contextual phenomenon. The analysis of our SLR results show that out of 22 case studies, 40% showed positive results. This may be due to the fact that data was collected at different stages of system development life cycle, organisations were of different sizes, projects were developing different types of systems, and the development methodologies employed were varied. Although the problem of the choice of research methodology mentioned above has long been identified as one of the reasons for lack of consistency in empirical findings [1,2], but to date we have not found a study that has utilized mixed method research approach to produce results with contextual details that is generalizable at the same time.

The nature of this impact is difficult to characterize due to the following reasons:

  1. The UI-SS relationship is very complex that involves too manyfactors thus making it difficult to analyze. Analysis of our SLR tells us that the choice of research method has an impact on the outcomes but the impact is not attributed only to the method used because there are other confounding factors (see Section 1).
  2. The UI-SS relationship can be studied from many different perspectives. The combination of these perspectives and different research methods used to study them will undoubtedly impact on the results obtained (see RQ3 in Section 7).
  3. The included studies were very diverse in their research designand instrument for measuring the UI-SS relationship and this heterogeneity prevented further analysis to be carried out about the impact of research method on the obtained results (see Section 2).

To explore the richness of the UI-SS relationship, using only one research method is not adequate. This calls for utilizing multi method approaches.

۸٫   Discussion

Overall, the results of our review present positive impact of user

 

Fig. 8. Stages of SDLC where user were involved.

 

involvement on system success. But the UI-SS relationship is not direct or binary. It is a multifaceted and convoluted concept where various confounding factors are playing their roles in contributing to system success. As mentioned in the Background Section, there are other secondary studies that have examined the available literature on UI-SS relationship. Our results are in accordance with the findings of these previous reviews and also enhance them by providing more coverage of the available literature by following the guidelines of SLR (see Table 2). Cavaye [2] provided review of 19 studies published from 1982 to 1992 and found that only 37% of the studies presented a positive UI-SS relationship. If we look closely at Fig. 6, we can see that in first decade (1980–۱۹۹۰) there were not that many studies showing positive results but from 1990 onwards the studies were showing more positive results. Cavaye [2] also acknowledged that UI-SS relationship has many

 

dimensions. Cavaye organized the confounding factors (referred to as ‘‘contingencies’’) into three categories i.e. Organizational factors (time for development, financial resources available, top management commitment), Project related factors (degree of task structure, project complexity, available technology, expected changes to bring in the system) and user related factors (willingness to participate, ability to participate, user characteristics and attitude).

He and King [3] performed a meta-analysis of 82 studies published from 1974 to 2007 and showed that user participation has statistically significant effect on the positive results. According to their study, researchers use different factors for measuring the outcomes of the user participation. They have classified those factors into two major categories i.e. attitudinal/behavioral outcomes (user satisfaction, user intention, system use) and productivity outcomes (individual impact, team performance, organizational impact, project quality, project success). Their meta-analysis has shown that there are stronger positive results for attitudinal/ behavioral outcomes than the productivity and that the psychological dimension of user involvement and participation has been more under the focus by the researchers when measuring system success. Our results show a similar pattern in Table 6, where ‘user system satisfaction’ has been used as a top factor for measuring the system success. He and King [3] suggested that user participation should be strategy driven based on the ultimate goal to achieve during system development. If user satisfaction and acceptance is the main objective then user participation should be managed and designed to induce more psychological involvement of the users. But if the productivity and quality of the system is the focus then user participation should be designed to facilitate and increase the domain knowledge of the development team. In Fig. 9 our results confirm and enhance the results of He and King [3] by showing that user involvement or participation can lead to system success indirectly by achieving the benefits from various perspectives as shown in Table 6. However, there are various additional factors (such as degree and level of involvement) that have to be taken into account while managing effective user involvement or participation.

Based on our findings to answer the research questions we were able to analyze the results for the following two purposes that will be discussed in more details below:

Development of the list of factors for effective management of user involvement. This would help the practitioners in designing more effective strategy for user involvement.

Identification and description of the factors causing the conflicting results in previous research. This would help the future research in careful design of the research method to further examine UI-SS relationship.

۸٫۱٫ Factors for effective management of user involvement

It is apparent from the literature that it is not enough just to involve the users in SDLC but this involvement or participation has to be effectively managed to achieve its intended objectives and the desired benefits. In this section we will outline the result of our synthesis from answering the research questions for effective management of user involvement.

۸٫۱٫۱٫ Identifying users for involvement and participation

Identification of the right type of users, who will be involved, and who will participate are important factors (see Findings: RQ2). Before the selection process, the users are to be identified and the concept of ‘user’ is to be clearly understood as it has many interpretations. According to Eason [37], there are three categories of users: primary, secondary and tertiary. ‘‘Primary users are those likely to be frequent hands-on users of the system; secondary users are occasional users or those who use the system through an intermediary; tertiary users are those affected by the introduction of the system, or who will influence its purchase’’. Damodaran [33] suggests further consideration about the types of users that should be involved: ‘‘‘first level’ users or ‘end users’, who will interact directly with computer terminals to help them to perform their work, will have different interests from those users who will only utilise printed output

or manage the primary users.’’

Once the users and their categories are identified various factors are to be considered for the selection of the users for involvement and participation. In line to the concept of stakeholder analysis [38], there is a need of user analysis, which would require answer to the following questions [33]: ‘‘Who will be affected? Is it possible to identify just two or three main user categories? What are the characteristics of people in each user category? What are the characteristics of the task performed by each user category? What do different users like and dislike about their jobs? How are the different users likely to react to the system?’’

۸٫۱٫۲٫ Perspective for user involvement

Perspective of user involvement is one of the most important factors as it would define the goals, objectives, needs and desired benefits that management wants to achieve for their system. Our analysis of literature identified five major perspectives for user involvement: Psychological, Managerial, Methodological, Political and Cultural (see Findings: RQ3). Tables 6–۸ presented the benefits, problems and challenges associated with user involvement in a particular perspective. We believe these perspectives present a new lens for practitioners to use when considering the problems, challenges and benefits for effective management of user involvement. Furthermore, in future studies a new approach to measure or study the UI-SS relationship can take into account the categorized benefits presented in Table 6. That is, one can select a specific perspective for the study and then use the listed benefits, problems, and challenges under that perspective for data collection.

۸٫۱٫۳٫ Degree and level of user involvement

Involving users is time and resource intensive and requires careful planning, execution and management. Increasing the degree and level of user involvement requires more careful planning and management for both the users and the developers. It is important to know what should be the intensity of their involvement for the corresponding desired benefits. So, the relationship between the desired benefits and the degree and level of user involvement should be carefully analyzed at the outset. Also these decisions will have an impact on which stages of SDLC users should be involved and what is the intensity of their involvement at each stage (see Findings: RQ8). We believe that our classification of desired benefits according to the perspectives (presented in Table 6) should shed light on this type of decisions.

۸٫۱٫۴٫ Stages of SDLC

Our analysis reveals that the stage(s) of SDLC for user involvement depends on the goals and perspective of user involvement. For example, to achieve the benefits in methodological and psychological perspectives, user involvement in requirements phase seems to be the most effective. For political and cultural benefits, users need to be involved in design and implementation phases (Table 6).

۸٫۱٫۵٫ Types of system being developed

Fig. 9. Factors found in SLR for UI-SS relationship.

It has been mentioned in various studies that the types of systems being developed would create a different context for user participation and therefore have different requirements regarding various aspects of user involvement [2,39]. User involvement takes a different form for development of modern day applications e.g. for online mobile applications [11], distributed collaborative application development environment [12], software requirements evolution [13], and in service oriented domain [41].

Fig. 9 is our attempt to encapsulate and bring together all the factors found in our SLR that contribute to the UI-SS relationship. Although these factors have been covered in the research literature but not all of them are supported with empirical evidence (see Section 11). This figure includes two distinct but related parts. The top part of the diagram illustrates that the UI-SS relationship is not direct. In order to achieve SS, we must have some ways of measuring the benefits UI produces. These benefits that in turn lead to system success are represented in the top part of the diagram based on 5 different perspectives. The bottom part of the diagram represents additional factors found in our SLR that also influence the UI-SS relationship. These factors were discussed in different parts of the paper and we have provided cross references for them in Fig. 9. We have also classified these additional factors based on the five perspectives identified in the paper and shown in the top part of this figure.

۸٫۲٫ Causes for conflicting results in empirical studies

Our systematic review of literature has provided insights into some of the causes of conflicts in the empirical findings of the last three decades. According to the analysis of our results, these conflicts are due to the following reasons:

۸٫۲٫۱٫ Different definitions and understanding of the concepts

The studies we reviewed referred to different established definitions of either user involvement or user participation but most of them were unclear about the concept. They used user involvement synonymously with ‘focus on users’, ‘consulting with end users’, ‘contacting with system users’ and ‘participation of users’ [۷]. The following three definitions were among the most cited in the 87 studies of our review, with the one by Hartwick and Barki receiving the highest citation.

  • Hartwick J. and Barki H. [14]: ‘‘a separation of the constructs of user participation (a set of behaviors or activities performed by users in the system development process) and user involvement (a subjective psychological state reflecting the importance and personal relevance of a system to the user).’’
  • Ives and Olson (S26): ‘‘user involvement is defined here as participation in the development process by a member or members of the target user group. User involvement can be examined on at least one of two different dimensions. First types of involvement, such as steering committees, representation on project teams, sign-offs on development stages, etc., can be examined as they relate to user attitudes and system use. These types will be referred to as mechanisms for implementing user involvement. The second dimension refers to process, at what stage in the system development life cycle is user involvement appropriate?’’
  • Doll and Torkzadeh (S40): ‘‘End-user involvement is defined as the extent to which the user engages in systems analysis activities such as project definition and logical design decisions. It is viewed solely in terms of the end-user rather than being a construct of relative effort or influence vis a vis a systems analyst. Of the three possible end-user involvement situations, only in the case of involvement with a systems analyst is it useful to think about involvement as user vs. analyst influence or communication relationship.’’

۸٫۲٫۲٫ Differences in research methods

According to the review of Ives and Olson [1], and Cavaye [2], one of the main reasons for inconsistent results is the differences in the choice of research design by researchers in their empirical investigation. Fig. 4 presented different methods used in investigating this complex and multi faceted phenomenon. Among them the method mostly used is survey followed by case studies. In first decade (1980–۱۹۸۹) surveys are dominating the research field as 80% of the studies. But they provided very conflicting results and in second and third periods gradually other qualitative research methods are adopted to study this phenomenon. Surveys are good for collecting structured data from a large population for testing a hypothesis. But as user involvement is known to be a complex concept that is not fully understood, surveys may not be able to capture the rich contextual details required to understand this phenomenon [2]. Although the problem of the choice of research method mentioned above has long been identified (since 1984) as one of the reasons for lack of consistency in empirical findings [1], but to date we have not found a study that has utilized mixed method research approach to produce results that cover the rich and contextual details and that could also be generalizable at the same time (see Findings: RQ9).

۸٫۲٫۳٫ Differences in data collection methods and measurement

instruments

The majority of the studies have used questionnaire surveys for their data collection. The second highest data collection method used was interviews. But the procedures they have followed differ significantly from each other on the following basis: Respondents’ sample type (managers, analysts, end users, and developers), size, type of Organizations and projects, phases of SDLC when data was collected, and Instruments for measurement. These differences undoubtedly have a profound impact on the consistency of the results obtained.

Another major difference among the studies is the instrument used for measuring user involvement or participation. The use of different constructs and variables for measurement has made it very difficult to consolidate the research findings or perform meta-analysis [2]. According to S34, there are five studies that provide the best measures for user involvement in system success: Franz and Robey (S5), Baroudi et al. (S2), Robey et al. [30], and Doll and Torkzadeh (S11, S46). We believe that future research should utilize and reuse the experiences reported in these five studies.

۸٫۳٫ Comparison of our SLR results with a recent mapping study

In parallel to our work, a recent study by Abelein and Paech [42] reviewed the UI-SS relationship and conducted a systematic mapping study using the guidelines of EBSE. Although the focus of the study is the same as ours, that is, to investigate the UI-SS relationship, but it differs from ours in following the systematic review process, formulation of research questions and analysis. Both studies have similar search string and sources for searching but the inclusion/exclusion criteria are not the same resulting in different set of papers. Only 41 papers out of 87 from our review are included in the 58 papers of that study. We were including all empirical papers that could provide answers to our research questions, whereas the study by Abelein and Paech was finding papers that would provide correlation data for meta-analysis. The major differences of our study with this mapping study are as follows:

  1. Both studies have different set of research questions for investigating UI-SS relationship.
  2. In secondary searches, we searched the major management, ISrelated journals, DBLP profiles of mostly cited authors, and performed comparisons with the existing literature reviews. Abelein and Paech do not conduct these steps.
  3. Our review included empirical papers for the duration of1980–۲۰۱۲, whereas the study of Abelein and Paech included the results from 1997 to 2012.
  4. Our review includes 87 studies whereas Abelein and Paech had58 papers included. The overlap of the included papers in both studies is 41.
  5. Our inclusion was based on the criteria of any empirical papersinvestigating UI-SS relationship; whereas the main focus of Abelein and Paech was to retrieve papers that provide them with correlation data for meta-analysis.
  6. In our study we did not differentiate between user involvementand user participation, due to the fact that most of the resulting studies considered them synonymous. But Abelein and Paech differentiated between the two concepts while performing their analysis.
  7. While analyzing UI-SS relationship we had three categories,positive, negative, and uncertain. Their study had only two categories positive and negative. Abelein and Paech considered the uncertain results as negative.

In spite of the differences in SLR process and analytical techniques used, the study of Abelein and Paech strengthens our results as follows:

  1. It confirms with meta-analysis that user involvement has positive impact on system success.
  2. It argues that user involvement is a complex socio-technicalphenomenon comprising of both human aspects and development aspects and hence it is difficult to measure.

۹٫   Limitations of SLR

Though we have succeeded to follow a very rigorous search strategy following the guidelines of EBSE to ensure the completeness of our sample, there would still be some papers that may not have been included in our data collection that we are not aware of due to their unavailability in electronic resources.

When we were composing our search strings based on the result of our pilot study and testing, we did not use an alternative term ‘‘engagement’’ for ‘‘involvement’’. This was found later at data extraction stage. With our rigorous secondary search strategy we believe that we have succeeded to compensate for this limitation but we concede that there may have been some papers that could have been included in our sample should we have used this term as well.

We eliminated the term ‘‘stakeholder’’ as a variant of the term ‘‘user’’ in our search string. This we did intentionally because not all stakeholders are considered users of the software product. Participants are often selected from among stakeholders based on the benefits they can bring to the system development because not all stakeholders carry equal relevance to the software product. According to Markus and Mao [27], from stakeholders we identify and select users for involvement and further among this set we select those who would be given the chance to actually participate in the development activities.

In developing the categories of perspectives (Section Findings: RQ3), the names given to the themes that emerged from our analysis were inspired by what has already been published in relevant papers as well as the intuition of the authors of this paper. We believe that these categories can be further validated by triangulation. An effective method for this task is card-sorting exercise employing independent participants from research or practitioner community [40].

۱۰٫   Conclusion

Although it was believed to be axiomatically true that user involvement brings about system success, this concept has been clouded due to plethora of the use of different definitions and conflicting results in empirical literature. Our SLR has painted a richer picture of this complex phenomenon by including many factors that play their role in the UI-SS relationship (Fig. 9). Our SLR enabled us to discover why the published empirical research has produced conflicting results. The aggregated knowledge of SLR will inform practitioners about various aspects of managing users, and their level and extent of involvement in different stages of software development under different conditions. For researchers, they can find the limitations of various research designs that are used for studying a complex multi variant phenomenon like the UI-SS relationship.

We found Systematic Review to be an effective choice for our exploration where the available literature is abundant. While comparing our results to the traditional literature reviews [1–۳], we could observe that their results were not as complete as our SLR due to the lack of rigor of systematic search and selection process

Table 9

Studies on agile.

Study ID UI-SS relationship Research method Decade of publication ERA rank
S14 Positive Case study ۱۹۹۰–۱۹۹۹ B
S20 Positive Case study ۲۰۰۰–۲۰۱۲ B
S23 Positive Case study ۲۰۰۰–۲۰۱۲ UR
S54 Uncertain Grounded theory ۲۰۰۰–۲۰۱۲ UR
S55 Positive Case study ۲۰۰۰–۲۰۱۲ UR

as well as undefined search boundaries. This is evident by the fact that 44 studies that were included in our review were missing in those reviews (Table 2).

۱۱٫   Future work

The main focus of our SLR was to explore the UI-SS relationship and we found that there is an extremely large body of research literature both empirical and non empirical available on this topic. However while analysing the included studies to answer our research questions we found some gaps in the empirical literature. Following is a list of concepts that are considered true axiomatically but are not explored well within empirical literature to provide convincing evidence for them and hence are open research areas.

  1. User identification and selection is considered very importantespecially in requirements analysis phase but the nature of its impact on the UI-SS relationship is not empirically investigated.
  2. How the different levels and degrees of user involvement affectthe outcomes of system development has not been empirically investigated.
  3. Different forms of cultural problems have been claimed to influence user involvement but the cultural perspectives of the UI-SS relationship is not investigated.
  4. A comparison of user involvement in different stages of SDLCand their outcome on system success needs to be further explored.
  5. A rigorous mixed method approach in research design and execution is needed to study the complex and multifaceted phenomenon of UI-SS relationship.

We were expecting to find more paper on these problems in the software engineering research literature but in our search results majority of work has been done by IS research community and is published mostly in management journals. User involvement and participation are not even included in Software Engineering Body of Knowledge (SWEBOK) [49] or Requirements Engineering Body of Knowledge (REBOK) [50]. Considering that user involvement is a mandatory part of Agile manifesto, our review surprisingly yielded only five studies (Table 9) that have investigated UI in the agile development environment and they are not homogenous in their focus so to support any generalisable conclusions. S14 aimed to provide a real case of agile customer engagement showing prerequisites, benefits, costs and risks in a software product setting. S20 provides an interesting comparison of user participation in agile and non-agile (traditional) projects. They found that the distinction between agile and non-agile projects with respect to the practice of user involvement is not clear and that active user involvement and participation throughout the SDLC can be successfully implemented in non-agile projects as well. S23 assessed the Participatory Design practices within agile methodologies by considering the level of involvement as informative, consultative, and participative. S54 performed a Grounded Theory research and presents the causes and consequences of lack of customer involvement on Agile projects and describe the Agile Undercover strategies used to overcome them. S55 has presented one aspect of user involvement i.e. communication in distributed agile development. Given that ‘‘user involvement’’ is fundamental in the agile methods, this should present a very rich area for future research under this topic.

We are using the findings of this SLR to analyze the results of our own ongoing research on user involvement. We have conducted two case studies of in-house software development in Australia. We are currently analyzing a huge qualitative data set of 57 interview transcripts. Our preliminary results indicate that there are major differences between user involvement in an ‘‘in-house’’ software development environment with internal users and developing software for external users. This is again a rich area for future research that seems to have somehow been neglected.

The results from both the SLR and the case studies will also contribute to the next step of the research project, which is to develop a user-centered method for service oriented software development. Our objective is to incorporate our findings of the traditional concepts of user involvement in the modern day service oriented system development. We are using online feedback of actual users of the service by employing social computing methods, and experiment with the level of impact on user satisfaction with the resulting system.

Appendix A. Included empirical studies

S1 J. Drew Procaccino, J.M. Verner, S.P. Overmyer, and M.E. Darter, ‘‘Case study: factors for early prediction of software development success,’’ Information and Software Technology, vol. 44, no. 1, pp. 53–۶۲, ۲۰۰۲
S2 J.J. Baroudi, M.H. Olson, and B. Ives, ‘‘An empirical study of the impact of user involvement on system usage and information satisfaction,’’ Communications of the ACM, vol. 29, no. 3, pp. 232–۲۳۸, ۱۹۸۶
S3 T. Heinbokel, S. Sonnentag, M. Frese, W. Stolte, and F.C. Brodbeck, ‘‘Don’t underestimate the problems of user centredness in software development projects there are many!,’’ Behaviour & Information Technology, vol. 15, no. 4, pp. 226–۲۳۶, ۱۹۹۶
S4 E.L. Wagner and S. Newell, ‘‘Exploring the importance of participation in the post-implementation period of an ES project: a neglected area,’’ Journal of the Association for Information Systems, vol. 8, no. 10, pp. 508–۵۲۴, ۲۰۰۷
S5 C.R. Franz and D. Robey, ‘‘Organizational Context, user involvement, and usefulness of information systems’’ Decision Sciences, vol. 17, no. 3, pp. 329–۳۵۶, ۱۹۸۶
S6 S. Kujala, ‘‘Effective user involvement in product development by improving the analysis of user needs,’’ Behaviour & Information Technology, vol. 27, no. 6, pp.

۴۵۷–۴۷۳, ۲۰۰۸

S7      M.J. Gallivan and M. Keil, ‘‘The user–developer communication process: a critical case study,’’

Information Systems Journal, vol. 13, no. 1, pp. 37–۶۸,

۲۰۰۳

S8 J.D. McKeen and T. Guimaraes, ‘‘Successful strategies for user participation in systems development,’’ Journal of Management Information Systems, pp. 133–۱۵۰, ۱۹۹۷

S9      S. Wilson, M. Bekker, P. Johnson, and H. Johnson, ‘‘Helping and hindering user involvement—a tale of everyday design,’’ ۱۹۹۷, pp. 178–۱۸۵

S10    D. Howcroft and M. Wilson, ‘‘Participation: ‘bounded freedom’ or hidden constraints on user involvement,’’ New Technology, Work and Employment, vol. 18, no. 1, pp. 2–۱۹, ۲۰۰۳

S11 G. Torkzadeh and W.J. Doll, ‘‘The test–retest reliability of user involvement instruments,’’ Information & Management, vol. 26, no. 1, pp. 21–۳۱, ۱۹۹۴

S12    M. Igbaria and T. Guimaraes, ‘‘Empirically testing the outcomes of user involvement in DSS development,’’

Omega, vol. 22, no. 2, pp. 157–۱۷۲, ۱۹۹۴

S13    S.T. Foster and C.R. Franz, ‘‘User involvement during information systems development: a comparison of analyst and user perceptions of system acceptance,’’ Journal of Engineering and Technology Management, vol. 16, no. 3, pp. 329–۳۴۸, ۱۹۹۹

S14    G.K. Hanssen and T.E. Fægri, ‘Agile customer engagement: a longitudinal qualitative case study’, in Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering, 2006, pp. 164–۱۷۳

S15 L.A. Kappelman and E.R. McLean, ‘‘Promoting information system success: the respective roles of user participation and user involvement,’’ Journal of

Information Technology Management, vol. 3, no. 1, pp.

۱–۱۲, ۱۹۹۲

S16                 P.J. Rondeau, M.A. Vonderembse, and T. Ragu-Nathan,

‘‘Investigating the Level of End-User Development and

Involvement Among Time-Based Competitors,’’ Decision

Sciences, vol. 33, no. 1, pp. 149–۱۶۰, ۲۰۰۲

S17    S. Kanungo and S. Bagchi, ‘‘Understanding user participation and involvement in ERP use,’’ Journal of

Management Research, vol. 1, no. 1, 2000

S18    N. Iivari, ‘‘‘Representing the User’in software development—a cultural analysis of usability work in the product development context,’’ Interacting with Computers, vol. 18, no. 4, pp. 635–۶۶۴, ۲۰۰۶

S19    R. Berntsson-Svensson and A. Aurum, ‘Successful software project and products: An empirical investigation’, in Proceedings of the 2006 ACM/IEEE International symposium on empirical software engineering, 2006, pp. 144–۱۵۳

S20 Z. Bakalova and M. Daneva, ‘A comparative case study on clients participation in a ’traditional’ and in an Agile software company’, in Proceedings of the 12th International Conference on Product Focused Software

Development and Process Improvement, 2011, pp. 74–

۸۰

S21 J. Heiskari and L. Lehtola, ‘Investigating the State of User Involvement in Practice’, in Asia-Pacific Software

Engineering Conference, 2009. APSEC’۰۹ pp. 433–۴۴۰

S22 S. Kujala, M. Kauppinen, L. Lehtola, and T. Kojo, ‘The role of user involvement in requirements quality and project success’, in 13th IEEE International Conference on Requirements Engineering, 2005 pp. 75–۸۴

 

S23 K. Kautz, ‘‘Participatory Design Activities and Agile

Software Development,’’ Human Benefit through the

Diffusion of Information Systems Design Science

Research, pp. 303–۳۱۶, ۲۰۱۰

S24 B. Johansson, ‘‘Diffusion of Open Source ERP Systems Development: How Users Are Involved,’’ Governance and Sustainability in Information Systems. Managing the

Transfer and Diffusion of IT, pp. 188–۲۰۳, ۲۰۱۱

S25 S. Pelayo, ‘‘From users Involvement to users’ needs understanding: a case study,’’ International Journal of Medical Informatics, vol. 79, no. 4, pp. e76–e82, 2010
S26 M.H. Olson and B. Ives, ‘‘User involvement in system design: an empirical test of alternative approaches,’’

Information & Management, vol. 4, no. 4, pp. 183–۱۹۵, ۱۹۸۱

S27 M.H. Olson and B. Ives, ‘‘Chargeback systems and user involvement in information systems-An empirical investigation,’’ MIS Quarterly, pp. 47–۶۰, ۱۹۸۲
S28 R.I. Mann and H.J. Watson, ‘‘A contingency model for user involvement in DSS development,’’ MIS Quarterly, pp. 27–۳۸, ۱۹۸۴
S29 C.L. Meador, M.J. Guyote, and P.G.W. Keen, ‘‘Setting priorities for DSS development,’’ MIS Quarterly, pp. 117– ۱۲۹, ۱۹۸۴
S30 K.B. White and R. Leifer, ‘‘Information systems development success: Perspectives from project team participants,’’ MIS Quarterly, pp. 215–۲۲۳, ۱۹۸۶
S31 P. Tait and I. Vessey, ‘‘The effect of user involvement on

system success: a contingency approach,’’ MIS Quarterly, pp. 91–۱۰۸, ۱۹۸۸

S32 A M.K. Baronas and M.R. Louis, ‘‘Restoring a sense of control during implementation: how user involvement leads to system acceptance,’’ MIS Quarterly, pp. 111– ۱۲۴, ۱۹۸۸
S33 M. Lawrence and G. Low, ‘‘Exploring individual user satisfaction within user-led development,’’ MIS Quarterly, pp. 195–۲۰۸, ۱۹۹۳
S34 H. Barki and J. Hartwick, ‘‘Measuring user participation, user involvement, and user attitude,’’ MIS Quarterly, pp.

۵۹–۸۲, ۱۹۹۴

S35 Y. Yoon, T. Guimaraes, and Q. O’Neal, ‘‘Exploring the factors associated with expert systems success,’’ MIS Quarterly, pp. 83–۱۰۶, ۱۹۹۵
S36 J.E. Hunton and J.D. Beeler, ‘‘Effects of user participation

in systems development: a longitudinal field experiment,’’ MIS Quarterly, pp. 359–۳۸۸, ۱۹۹۷

S37 J.D. McKeen, T. Guimaraes, and J.C. Wetherbe, ‘‘The relationship between user participation and user satisfaction: an investigation of four contingency factors,’’ MIS Quarterly, pp. 427–۴۵۱, ۱۹۹۴
S38 K.E. Emam, S. Quintin, and N.H. Madhavji, ‘‘User participation in the requirements engineering process: An empirical study,’’ Requirements Engineering, vol. 1, no. 1, pp. 4–۲۶, ۱۹۹۶
S39 J.D. Gould and C. Lewis, ‘‘Designing for usability: key principles and what designers think,’’ Communications of the ACM, vol. 28, no. 3, pp. 300–۳۱۱, ۱۹۸۵
S40 Doll, William J., and Gholamreza Torkzadeh. ‘‘A discrepancy model of end-user computing involvement.’’ Management Science vol 35, no. 10 (1989): 1151–۱۱۷۱
S41 S. Pekkola, N. Kaarilahti, and P. Pohjola, ‘‘Towards formalised end-user participation in information

(continued on next page)

  Systems, pp. 145–۱۶۶, ۱۹۹۶
S58 J. Hartwick and H. Barki, ‘‘Communication as a dimension of user participation,’’ Professional

Communication, IEEE Transactions on, vol. 44, no. 1, pp.

۲۱–۳۶, ۲۰۰۱

S59 Vadapalli, A., and Mone, M.A. Information technology project outcomes: User participation structures and the impact of organization behavior and human resource management issues. Journal of Engineering and Technology Management, 17, 2 (2000), 127–۱۵۱
S60 King, W.R., and Rodriquez, J.I. Participative design of strategic decision support systems: An empirical assessment. Management Science, 27, 6 (June 1981), 717–۷۲۶
S61 Carayon, P., and Karsh, B.T. Sociotechnical issues in the implementation of imaging technology. Behaviour & Information Technology, 19, 4 (2000), 247–۲۶۲
S62 Kim, E., and Lee, J. An exploratory contingency model of user participation and MIS use. Information and Management, 11, 2 (1986), 87–۹۸
S63 Barki, H., and Huff, S.L. Implementing decision support systems: Correlates of user satisfaction and system usage. INFOR, 28, 2 (May 1990), 89–۱۰۱
S64 Doll, W.J., and Torkzadeh, G. A congruence construct of user involvement. Decision Sciences, 22, 2 (Spring 1991), 443–۴۵۳
S65 Jarvenpaa, S.L., and Ives, B. Executive involvement and participation in the management of information technology. MIS Quarterly, 15, 2 (June 1991), 205–۲۲۷
S66 Allingham, P., and O’Connor, M. MIS success: Why does it vary among users? Journal of Information Technology, 7, 3 (1992), 160–۱۶۸
S67 Steinbart, P.J., and Accola, W. The effects of explanation type and user involvement on learning from and satisfaction with expert systems. Journal of Information

Systems, 8, 1 (Spring 1994), 1–۱۷

S68 Leonard-Barton, D., and Sinha, D. Developer–user interaction and user satisfaction in internal technology transfer. Academy of Management Journal, 36, 5 (1993), 1125–۱۱۳۹
S69 Wastell, D., and Sewards, A. An information system profile of the UK manufacturing sector. Journal of Information Technology, 10, 3 (1995), 179–۱۸۹
S70 Sioukas, A.V. User involvement for effective customization: An empirical study on voice network. IEEE Transactions on Engineering Management, 42, 1 (February 1995), 39–۴۹
S71 Hunton, J.E. Involving information system users in defining system requirements: The influence of procedural justice perceptions on user attitudes and performance. Decision Sciences, 27, 4 (Fall 1996), 647– ۶۷۱
S72 Hunton, J.E. User participation in defining system interface requirements: An issue of procedural justice. Journal of Information Systems, 10, 1 (Spring 1996), 27– ۴۷
S73 Hunton, J.E., and Price, K.H. Effects of the user participation process and task meaningfulness on key information system outcomes. Management Science, 43, 6 (June 1997), 797–۸۱۲
S74 Lu, H.P., and Wang, J.Y. The relationships between management styles, user participation, and system success over MIS growth stages. Information and Management, 32, 4 (April 1997), 203–۲۱۳

systems development process: bridging the gap between participatory design and ISD methodologies,’’ In Proceedings of the ninth conference on Participatory design: Expanding boundaries in design-Volume 1, pp.

۲۱–۳۰٫ ACM, 2006

S42 G.J. de Vreede, S.O. den Hengst, and H.G. Sol, ‘‘Facilitating user involvement in information system design and development with GSS: the organized crime case,’’ In Proceedings of the 1995 ACM SIGCPR conference on Supporting teams, groups, and learning inside and outside the IS function reinventing IS (pp. 81–۹۱). ACM

S43    D. Robey and D. Farrow, ‘‘User involvement in information system development: A conflict model and empirical test,’’ Management Science, pp. 73–۸۵, ۱۹۸۲

S44    L.A. Kappelman, ‘‘Measuring user involvement: A diffusion of innovation perspective,’’ ACM SIGMIS Database, vol. 26, no. 2–۳, pp. 65–۸۶, ۱۹۹۵

S45 K. Amoako-Gyampah and K.B. White, ‘‘User involvement and user satisfaction:: An exploratory contingency model,’’ Information & Management, vol. 25, no. 1, pp.

۱–۱۰, ۱۹۹۳

S46    W. Doll and G. Torkzadeh, ‘‘The measurement of enduser software involvement,’’ Omega, vol. 18, no. 4, pp.

۳۹۹–۴۰۶, ۱۹۹۰

S47    J. Hartwick and H. Barki, ‘‘Explaining the role of user participation in information system use,’’ Management Science, pp. 440–۴۶۵, ۱۹۹۴

S48    P. Kawalek and T. Wood-Harper, ‘‘The finding of thorns: user participation in enterprise system implementation,’’ ACM SIGMIS Database, vol. 33, no. 1, pp. 13–۲۲, ۲۰۰۲

S49    M. Newman and F. Noble, ‘‘User involvement as an interaction process: a case study,’’ Information Systems Research, vol. 1, no. 1, pp. 89–۱۱۳, ۱۹۹۰

S50 W.T. Lin and B. Shao, ‘‘The relationship between user participation and system success: a simultaneous contingency approach,’’ Information & Management, vol.

۳۷, no. 6, pp. 283–۲۹۵, ۲۰۰۰

S51 W.J. Doll, ‘‘Encouraging user management participation in systems design,’’ Information & Management, vol. 13,

  1. ۱, pp. 25–۳۲, ۱۹۸۷

S52 S.R. Hawk and B.L. Dos Santos, ‘‘Successful system development: the effect of situational factors on alternative user roles,’’ IEEE Transactions on Engineering Management, vol. 38, no. 4, pp. 316–۳۲۷, ۱۹۹۱

S53    S. Gasson, ‘‘User involvement in decision-making in information systems development,’’ In Proceedings of

۱۸th IRIS. Gjern Denmark: IRIS Association 1995

S54    R. Hoda, J. Noble, and S. Marshall, ‘‘Agile undercover: when customers don’t collaborate,’’ Agile Processes in Software Engineering and Extreme Programming, pp.

۷۳–۸۷, ۲۰۱۰

S55 M. Korkala, M. Pikkarainen, and K. Conboy, ‘‘Distributed agile development: A case study of customer communication challenges,’’ Agile Processes in Software

Engineering and Extreme Programming, pp. 161–۱۶۷,

۲۰۰۹

S56    G. Bjerknes and T. Bratteteig, ‘‘User participation and democracy: A discussion of Scandinavian research on system development,’’ Scandinavian Journal of

Information Systems, vol. 7, pp. 73–۷۳, ۱۹۹۵

S57    N. Saleem, ‘‘An empirical test of the contingency approach to user participation in information systems development,’’ Journal of Management Information

 

S75 Choe, J.M. The effects of user participation on the design of accounting information systems. Information and Management, 34, 3 (October 1998), 185–۱۹۸

S76 Zeffane, R., Cheek, B., and Meredith, P. Does user involvement during information systems development improve data quality? Human Systems Management, 17,

۲ (۱۹۹۸), ۱۱۵–۱۲۱

S77 Foster, S.T., Jr., and Franz, C.R. On the difference between information systems’ users and analysts: Managing perceptions of systems quality. Journal of Quality Management, 3, 1 (1998), 63–۷۷

S78    Doll, W.J., and Deng, X. The collaborative use of information technology: End-user participation and system success. Information Resources Management Journal, 4, 2 (April–June 2001), 6–۱۶

S79    Brodbeck, F.C. Communication and performance in software development projects. European Journal of Work & Organizational Psychology, 10, 1 (2001), 73–۹۴

S80          Yeh, Q.J., and Tsai, C.L. Two conflict potentials during IS development. Information and Management, 39, 2

(December 2001), 135–۱۴۹

S81 Aladwani, A.M. IT planning effectiveness in a developing country. Journal of Global Information Technology Management, 4, 3 (2001), 51–۶۵

S82    Palanisamy, R. Strategic information systems planning model for building flexibility and success. Industrial Management & Data Systems, 105, 1 (2005), 63–۸۱

S83 Palanisamy, R., and Sushil, J.L. Empirically testing the relationships between user involvement, information waste, and MIS success. Journal of Services Research, 1, 1

(April–September 2001), 70–۱۰۳

S84 Palanisamy, R., and Sushil, J.L. User involvement in information systems planning leads to strategic success:

An empirical study. Journal of Services Research, 1, 2 (October 2001), 125–۱۵۷

S85    Lawrence, M., Goodwin, P., and Fildes, R. Influence of user participation on DSS use and decision accuracy. Omega, 30, 5 (2002), 381–۳۹۲

S86    Terry, J., and Standing, C. Do project manager’s utilise potential customers in e-commerce developments? In Proceedings of the Informing Science and Information

Technology Education Joint Conference 2004. Santa

Rosa, CA: Informing Science Institute, 2004, pp. 663–۶۷۲

S87 Wu, J.B., and Marakas, G.M. The impact of operational user participation on perceived system implementation success: An empirical investigation. Journal of Computer Information Systems, 46, 5 (2006), 127–۱۴۰

Appendix B. Quality assessment checklist

The scoring to the checklist was based on the three potential answers to the questions; yes = 1, partial = 0.5 and no = 0. If any of the criteria was not applicable on any study then it was excluded from evaluating for only that particular study. The studies that scored less than 50% in quality assessment were excluded as they were not providing the basic information about their research methodology.

# Generic  
۱ Are the aims clearly stated? YES/NO
۲ Are the study participants or observational units adequately described? YES/NO/

PARTIAL

  • Was the study design appropriate with YES/NO/ respect to research aim?              PARTIAL
  • Are the data collection methods YES/NO/ adequately described? PARTIAL
  • Are the statistical methods justified by YES/NO the author?
  • Is the statistical methods used to analyze YES/NO the data properly described and referenced?
  • Are negative findings presented? YES/NO/

PARTIAL

  • Are all the study questions answered? YES/NO
  • Do the researchers explain future YES/NO implications?

Survey

  • Was the denominator (i.e. the YES/NO population size) reported?
  • Did the author justified sample size? YES/NO
  • Is the sample representative of the YES/NO population to which the results will generalize?
  • Have ‘‘drop outs’’ introduced biasness on YES/NO/NOT result limitation? APPLICABLE

Experiment

  • Were treatments randomly allocated? YES/NO
  • If there is a control group, are YES/NO participants similar to the treatment group participants in terms of variables that may affect study outcomes?
  • Could lack of blinding introduce bias? YES/NO
  • Are the variables used in the study YES/NO adequately measured (i.e. are the variables likely to be valid and reliable)?

Case study

  • Is case study context defined? YES/NO
  • Are sufficient raw data presented to YES/NO provide understanding of the case?
  • Is the case study based on theory and YES/NO linked to existing literature?
  • Are ethical issues addressed properly YES/NO

(personal intentions, integrity issues, consent, review board approval)?

  • Is a clear Chain of evidence established YES/NO/ from observations to conclusions? PARTIAL

Experience report

  • Is the focus of study reported? YES/NO
  • Does the author report personal YES/NO observation?
  • Is there a link between data, YES/NO/ interpretation and conclusion?     PARTIAL
  • Does the study report multiple YES/NO experiences?

References

Manage. Sci. 30 (1984) 586–۶۰۳.

[۱] https://docs.google.com/file/d/0B8AfnftYIxYDaWphMEVpZG5MclE/ edit?usp=sharing.

[۲] ERA (http://www.arc.gov.au/era/) is administered by Australian Research Council which aims to identify and promote excellence across the full spectrum of research activity in Australia’s higher education institutions. It also provides a comprehensive overview of the quality of research undertaken in higher education institutions across the country, in an international context. In ERA ranking, the journals and conferences are ranked into five categories based on their quality (A*, A, B, C, UR (unranked)).

[۳] http://en.wikipedia.org/wiki/Citation_analysis.