Filetype PDF | Posted on 13 Sep 2022 | 4 years ago
Authentication
digital resources
258x Filetype PDF File size 2.05 MB Source: pdfs.semanticscholar.org
Proceedings of the 50th Hawaii International Conference on System Sciences | 2017
Risk –Informed Decision Making in Information System Implementation
Projects: Using Qualitative Assessment and Evaluation of Stakeholders’
Perceptions of Risk
Monica Schurr Manuel De Tuya Kathryn Noll
University at Albany University at Albany Rensselaer Polytechnic Institute
mlmeissner@albany.edu mdetuya@albany.edu tknoll12@gmail.com
Abstract decision making and risk response benefited a
manufacturing organization during and after the
The successful implementation of a new software implementation of a new department-wide software
system at any organization requires identification system. The purpose of the system was to move the
and management of risks as well as insight into the organization out of a paper-based manufacturing
decision-making process throughout the information process and into managing their production process
system lifecycle. Risk assessment of software systems via automated workflows able to control the
aids in planning, implementation and adoption stages execution of the manufacturing steps. The
and helps identify potential problems before they organization implementing the system could be
occur. This study utilized a qualitative case study characterized as highly concentrated on achieving
method and an interview design for data collection to excellence in their core competencies. Such core
competencies, as identified by company executives,
gather, organize and make sense of key stakeholders’ fell within the quality assurance and manufacturing
perceptions of risk for decision making in the areas, with the latter being the actual owner of the
implementation of a new department-wide system, making the project a department-wide
computerized system. Top stakeholder risks identified implementation. Nevertheless, the scope of the
include executive sponsorship support; adoption of system required the establishment of a cross-
the new technologies and processes; and functional implementation team to ensure that cross-
interoperability. The results of the analysis of departmental processes were considered when
perceptions of risks allowed the organization and the configuring the software solution.
team responsible for the implementation of the new The risk assessment for this study identified risks
system to make decisions about mitigating strategies associated with the new information system,
aligned with stakeholders’ expectations; forecast
potential issues within the implementation timeline hereinafter referred to as “the system”. Risks
based on activities associated with identified risks; associated with the system were based on perceptions
and make implementation and process decisions from areas of business such as Manufacturing,
based upon the risk assessment. This study extends Information Technology (IT), Quality Assurance
the research on IT risk management and decision (QA), Supply Chain, Process Controls, and
making by demonstrating the utility and efficacy of a Management. Traditionally, risk assessments for
qualitative case study method for eliciting the software implementation projects are performed
information needed from stakeholders in order to utilizing a variety of quantitative methods. In the case
make decisions regarding system implementation, of the organization being studied, there was a lack of
specifically in an organization that lacks the expertise in performing such assessments, in
appropriate risk management maturity level to particular for software projects. Using a qualitative
conduct an exhaustive quantitative analysis of risks method allowed the implementation team access to
associated with the project. key organizational representatives of the areas being
affected by the system.
1. Introduction This paper is organized into six sections: section
one includes an introduction to the paper in general
and this information system implementation project
This paper describes how a qualitative approach specifically; section two provides a brief literature
for assessing and evaluating risks in order to inform review; section three describes the methodology;
URI: http://hdl.handle.net/10125/41903
ISBN: 978-0-9981331-0-2
CC-BY-NC-ND 6120
section four presents results; section five analyzes The interviews for this paper were conducted
and discusses the interviews and identified risks, and between the end of the planning phase and the
details the risk-informed decision making process beginning of the implementation phase.
that resulted from this work; section six presents
limitations; and section seven presents contributions 2. Literature Review
and concluding remarks.
2.1. Risk Management for IT Projects
1.1 Description of the Information System Risks are classified as events that have adverse
Implementation Project outcomes. Risk management is a process involving
assessment, response and mitigation that can help
The main goal of the project was to improve prevent risk from occurring, as well as minimize
productivity and reduce cycle-time in the total time to damage and contain the cost of recovering from risk,
produce a manufacturing order. Automating the if risk does occur [1]. While risk can never be
process of controlling manufacturing records would entirely eliminated from a system, performing risk
optimize production activities within the organization assessment aids in identifying current or potential
and most likely bring additional synergies when risks associated with the implementation and
interacting with external manufacturers. The potential operation of a computerized system in a given
benefits identified during the development of the organization [2]. Additionally, it can provide
business cases were, among others, increased strategies to manage identified risks at a level that is
productivity, savings in labor costs, enhanced acceptable for the organization [3]. Risks are
management capabilities, shorter reaction time to assessed by examining magnitude and likelihood [1,
changing market conditions and higher availability of 3, 4], and risk response involves the organization
manufacturing information throughout the creating and implementing both preventative and
organization. corrective controls to ensure risk is minimized [5].
The proposed approach was to find a best of breed Additionally, risk mitigation acts to introduce
solution that could be integrated into the controls that reduce potential risks within a system, to
organization’s current technology landscape and long address risks and generate solutions to reduce and
term business and IT strategy. A transformative resolve threats [6].
initiative like this required the establishment of a Risk management within IT systems is vital to
governance body that included members of the ensure that systems operate within specific
leadership team acting as executive sponsors as well performance and computational accuracy thresholds
as active members of a steering committee. A project previously agreed upon in the form of user
manager from the IT department was in charge of the requirements and made official via Service Level
formal management of the project across the areas of Agreements (SLAs) [2, 3]. Generally speaking,
the organization needed in the definition and managing risks in a software implementation project
execution of the project deliverables. is a three phase process. Each of these phases may
As part of the management of the project, a present different types of risks and, accordingly,
comprehensive project timeline was produced, which different methods for managing them [7] .
listed a 17-month implementation strategy that The objective of a risk analysis and identification
included the definition of user and functional process is to provide information to facilitate the
requirements, definition of interfaces with other decision making process related to the
existing applications, unit, system and integration implementation of risk management strategies
testing phases and a final user acceptance testing whether it is acceptance, elimination or reduction [8].
phase followed by a month-long deployment into the Traditionally, risk assessments for software
production environment. The effort was divided into implementation projects rely heavily on a variety of
phases as per project management best practices: quantitative methods [9-11] that concentrate on the
a planning phase where high-level risk analysis and mitigation efforts to project-specific
requirements were gathered, vendors were deliverables or processes, which lead to a project-
screened and selected and budgets were specific decision-making modeling [12].
submitted for approval Nevertheless, software implementation projects
an implementation phase to design and produce business-specific (operational) risks that
configure the system should be quantified and, if needed, managed [9, 13].
a testing and deployment phase To that end, research has demonstrated that involving
6121
business subject matter experts (SMEs) positively vague, broad, and general statements/indicators [21,
impacts the performance of the implementation team 25-27].
and creates a sense of ownership for the SMEs when In applying this educational assessment and
they perceive the system as their own creation [14]. evaluation perspective to the assessment of risk in an
Appropriately addressing user (SMEs) perceptions of IT project, the indicator of the presence of learning
risk have been linked to increased levels of alignment (i.e. learning goal) can instead be framed in terms of
across the business as well as higher levels of an indicator of presence of risk (or, as the case may
organizational awareness [14]. be, the perceived presence of risk). Furthermore, the
The field of risk assessment and decision making concept of coarse-grained and fine-grained
is multifaceted and the processes multidisciplinary, information can be applied in terms of broad
which must be taken into account when considering a indicators of risk (e.g. issues with document
scientific platform and/or framework for risk [15]. maintenance) that can be broken down into more
Many theories explaining risk and decision making specific indicators (e.g. issues with record storage,
form the foundation of quantitative studies for risk ease of access, maintaining paper records and need
analysis and management, including decision theory, for backups, among others). As in the field of
the behavioral view of risk, and the real options view education, collecting this information at such a fine-
of risk [16-18]. While many consider quantitative grained level can inform decisions (what we will call
risk assessment (QRA) the method for estimating and or consider a form of risk evaluation) as much as the
quantifying risk, one must also consider that “societal actual actions. An example of this is users driving
risk decision making” – which stems from identifying organizational change management, as discussed in
such risks – requires consideration of stakeholders’ the next section.
understandings as well as contextual factors [15]. A
qualitative risk assessment targets the elicitation of 2.3. Users Driving Organizational Change
such important information (i.e. the answers to Management
“what” and “how” questions) and thus provides
pragmatic grounds for an exploratory method, which A determining success factor for the
could also lay the groundwork for theory implementation of computerized systems is the level
development [19]. of readiness achieved by the organization prior to
deploying the new technology [28, 29]. Such a state
of readiness is achieved by the appropriate planning
2.2. Assessing and Evaluating Risk and execution of an organizational change
management process [30], which consists of making
Understandings from the field of education with the organization aware of the change, educating users
regards to assessment and evaluation can provide a and secondary stakeholders on the consequences of
theoretical framework for the development of a the change and how to deal with it and creating the
qualitative interview protocol, the collection of data corresponding mechanisms so that the new status is
on specific risk indicators (assessment) as well as the adopted as seamlessly as possible [28, 30].
use of the information gathered from these qualitative A specific approach for facilitating organizational
interviews to inform decision making on risk change consists of involving non-supervisor members
management, mitigation, and reduction (evaluation). of the organization in a semi-crowdsourcing mode of
One approach in education is to separate the problem solving, also known as participative
concept of assessment from testing and grading, and leadership [31]. Research has positioned
understand it as the extent to which one has attained a participative leadership not only as a generator of
learning goal; and evaluation can be thought of as trust, but as a driver for enhanced organizational
applying that assessment information to inform and performance [31] and it is also positively influenced
make decisions [20-22]. For purposes of clarity and by higher degrees of information sharing from
precision when measuring attainment, broad learning supervisors [32]. This approach provides subject
goals can be written at very specific levels. matter experts, acting as subordinates of the project
Specifying (learning) indicators at a fine-grained leadership team, with intrinsic motivation for finding
level as opposed to a coarse-grained level [23, 24] innovative and effective solutions for specific
allows for collection of useful information and thus organizational needs [30, 33].
clear and specific measurement of attainment The inclusion of users (Subject Matter Experts or
(assessment) as well as actionable evaluation (using Stakeholders) in the risk management process should
the information to inform decisions) and eliminates provide a better understanding of perceived risks
the potential for confusion that is wrought with within the organization [15]. Such risks and their
6122
corresponding mitigation could either hinder or on pragmatic grounds as an opportunity to involve
promote the organizational change management future end users of the system in the identification of
process required for the successful implementation risks while gathering their perceptions of the project
and eventual maintenance of a transformative at large. In addition, the lack of in-house knowledge
computerized system [11,15]. Applying an on performing quantitative risk assessment and the
assessment and evaluation approach in this context need to keep the timeline unchanged made the
helps to frame the change management process in method ideal for this particular project.
terms of specific intended outcomes for said
processes. Employing assessment at a fine-grained Participant selection. Interviewees were selected
level allows identification of specific risks; based on level of involvement with the
employing evaluation allows us to use the implementation of the system into the company. The
information that results from the assessment to make total group (N = 27) was selected from
decisions in terms of implementation and manufacturing (N = 11), supply chain (N = 1), IT (N
maintenance. One can then gather information as to = 5), quality assurance (QA) (N = 4), process
whether the intended outcomes have occurred by sciences (N = 2), and management (N = 4), to
using evaluation techniques at the program level encompass a wide range of perceptions associated
(see, for example, [34] for a discussion of standards with implementation of the system.
for program evaluation).
Data collection approach. This case study
research utilized an interview design for accessing
and collecting data. Interviews lasted 30 minutes,
3. A Qualitative Approach
during which time interviewees’ perceptions of risk
associated with implementation of the system were
3.1 Case Study Research Method recorded.
The interviews were conducted in a semi-
Case studies facilitate the gathering of information structured manner, with a general set of questions
necessary for making decisions, as well as focusing prepared, but improvisation was used to obtain more
on the factors that influenced decisions within each specific information based on the subjects’
case and then comparing such factors in order to test knowledge and experience with the system.
existing theoretical constructs and relationships [35]. For each department, a different set of exploratory
Traditionally, for software implementation projects, “what” and “how” questions were used in order to
risk management is performed by analyzing elicit the thinking and opinions of each respective
indicators related to the development or group [19]. Sample questions included: “What are the
implementation process – requirements complexity, main risks you feel the system could generate for
software size, computational complexity and manufacturing that would interfere with the benefits
interfacing level, among others. By using an of the system?” (Manufacturing); “How will
interview design, indicators of risk (via perceptions) implementation of the new system affect the
can be gathered from a broad range of stakeholders, functionality of current systems?” (IT); “What could
both technical and non-technical, to gain a better be the risks if data integrity is compromised?”
understanding of concerns regarding the effects of the (Supply chain); “How could implementing the
new software on existing processes and computerized system affect compliance?” (Quality Assurance);
systems. More specifically, through semi-structured “What are the potential impacts or risks if the system
interviews [36], the level of flexibility facilitates is not accurate?” (Process sciences).
descriptive responses that allow the researcher to
develop detailed descriptions, integrate multiple Coding and data processing methods. Interview
perspectives, develop holistic descriptions and frame transcripts were reviewed after each interview to
hypotheses for quantitative research [37]. ensure that the proper meaning of the interviewees’
Furthermore, qualitative interview techniques lend responses were recorded. These interview transcripts
themselves to: the ability to generate reliable and were broken into smaller units, based on categories
valid data and reduce bias, such as via consensual created to reflect the main ideas of the responses
qualitative research methods [38]; inductive and gathered. For each interview question asked, the
deductive methods of analysis [39]; quantification for number of people surveyed was recorded; this was
further analysis [40], as well as laying the followed by responses to the question in unitized,
groundwork for theory development [19]. The coded form, listing category and subcategory, as well
qualitative approach for this study was chosen based as the participant’s identification number and
6123
no reviews yet
Please Login to review.
