NATIONAL SOFTWARE TESTING GUIDELINE 2019
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
NATIONAL SOFTWARE TESTING GUIDELINE
2019National Software Testing Guideline
Table of Contents
TERMS AND DEFINITIONS ...................................................................................................................... 4
EXECUTIVE SUMMARY ......................................................................................................................... 19
1 INTRODUCTION ............................................................................................................................ 20
1.1 Authority............................................................................................................................... 20
1.2 Scope ..................................................................................................................................... 20
1.3 Benefits of a National Software Testing Guideline ............................................................. 20
1.4 Objectives of The National Software Testing Guideline ..................................................... 21
2 THE SOFTWARE DEVELOPMENT LIFECYCLE (SDLC) ..................................................................... 21
2.1 The Software Testing Life Cycle ........................................................................................... 23
2.2 Software Testing Principles .................................................................................................. 24
2.3 Project Organisation Structure ............................................................................................ 24
2.4 Minimum Requirements to become a Software Test Professional .................................... 26
3 SOFTWARE TESTING GUIDELINE APPROACH............................................................................... 26
3.1 Organisational Test Process ................................................................................................. 27
3.1.1 Organisational Test Specifications ....................................................................................... 27
3.1.1.2 Purpose................................................................................................................................. 28
3.1.2 Practical Guideline................................................................................................................ 29
3.2 Test Management Processes ............................................................................................... 30
3.2.2 Practical Guidelines .............................................................................................................. 31
3.3 Test Planning Process ........................................................................................................... 31
3.3.1 Test Planning Input Requirements ...................................................................................... 31
3.3.2 Practical Guideline................................................................................................................ 36
3.3.3 Information Items ................................................................................................................ 40
.......................................................................................................................................................... 41
3.4 Test Monitoring and Control Process .................................................................................. 42
3.4.1 Overview ............................................................................................................................... 42
3.4.2 Purpose ................................................................................................................................. 43
3.4.3 Outcomes.............................................................................................................................. 43
3.4.4 Practical Guideline................................................................................................................ 43
3.4.5 Information Items ................................................................................................................ 44
3.5 Test Completion Process ...................................................................................................... 45
3.5.1 Overview ............................................................................................................................... 45
3.5.2 Purpose ................................................................................................................................. 45
3.5.3 Outcome ................................................................................................................................ 45
3.5.4 Activities and Tasks ............................................................................................................... 45
1|PageNational Software Testing Guideline
3.5.5 Practical Guideline............................................................................................................... 45
3.5.6 Information Items................................................................................................................ 46
3.6 Dynamic Testing Process ...................................................................................................... 46
3.6.2 Test Design and Implementation Process ........................................................................... 47
3.6.3 Purpose ................................................................................................................................. 47
3.6.4 Practical Guidelines .............................................................................................................. 48
3.6.5 Information Items ................................................................................................................ 49
3.7 Test Environment Set-Up and Maintenance ....................................................................... 49
3.7.1 Overview................................................................................................................................ 49
3.7.2 Purpose .................................................................................................................................. 50
3.7.3 Outcomes............................................................................................................................... 50
3.7.4 Activities and tasks................................................................................................................ 50
3.7.5 Practical Guideline ................................................................................................................ 50
3.7.6 Information Items ................................................................................................................. 50
3.8 Test Execution Process ......................................................................................................... 51
3.8.1 Overview ............................................................................................................................... 51
3.8.2 Purpose ................................................................................................................................. 51
3.8.3 Outcomes............................................................................................................................... 51
3.8.4 Activities and tasks................................................................................................................ 51
3.8.5 Practical Guideline ................................................................................................................ 51
3.8.6 Information Items ................................................................................................................. 52
3.9 Test Incident Report Process ............................................................................................... 52
3.9.1 Overview................................................................................................................................ 52
3.9.2 Purpose .................................................................................................................................. 52
3.9.3 Outcomes............................................................................................................................... 52
3.9.4 Activities and tasks................................................................................................................ 52
3.9.5 Practical Guideline ................................................................................................................ 53
3.10 Software Testing Practical Steps .......................................................................................... 53
4 LEVELS OF TESTING ....................................................................................................................... 58
4.1 Unit Testing........................................................................................................................... 59
4.1.1 Purpose ................................................................................................................................. 59
4.2 Integration Testing ............................................................................................................... 59
4.2.1 Purpose ................................................................................................................................. 59
4.3 System Testing...................................................................................................................... 59
4.3.1 Purpose ................................................................................................................................. 60
4.4 Acceptance Testing............................................................................................................... 60
2|PageNational Software Testing Guideline
4.4.1 Purpose ................................................................................................................................. 60
4.5 Software Test Automation Tools ......................................................................................... 61
5 TEST TECHNIQUES ........................................................................................................................ 62
5.1 Specification-Based Testing Techniques (Black-Box Testing Techniques) ......................... 62
5.2 Structure-Based Testing Techniques (White Box Testing Techniques) .............................. 62
5.2.1 White Box Testing Techniques ............................................................................................. 62
5.3 Experience-Based Testing Technique .................................................................................. 63
6 TEST DOCUMENTATION ............................................................................................................... 63
6.1 Overview ............................................................................................................................... 63
6.2 Organisational Test Policy Documentation ......................................................................... 64
6.2.1 Test Policy Document Template .......................................................................................... 64
6.3 Organisational Test Strategy Documentation ..................................................................... 65
6.3.1 Organisational Test Strategy Template ............................................................................... 65
6.4 Test Plan Documentation ..................................................................................................... 67
6.4.1 Test Plan Template ............................................................................................................... 67
6.5 Test Status Report Documentation ..................................................................................... 70
6.5.1 Test Status Report Template ............................................................................................... 70
6.6 Test Completion Report Documentation ............................................................................ 71
6.6.1 Test Completion Report Template....................................................................................... 71
6.7 Test Design Specification Documentation ........................................................................... 73
6.7.1 Test Design Specification Template ..................................................................................... 73
6.8 Test Case Specification Documentation .............................................................................. 74
6.8.1 Test Case Specification Templates ....................................................................................... 74
6.9 Test Procedure Specification Documentation ..................................................................... 75
6.9.1 Test Procedure Specification Template ............................................................................... 75
6.10 Test Data Requirements Documentation ............................................................................ 76
6.10.1 Test Data Requirements Template ...................................................................................... 76
6.11 Test Environment Requirements Documentation .............................................................. 76
6.11.1 Test Environment Requirements Template ........................................................................ 77
6.12 Test Execution Log Documentation ..................................................................................... 77
6.12.1 Test Execution Log Template ............................................................................................... 77
6.13 Incident Report Documentation .......................................................................................... 77
6.13.1 Incident Report Template .................................................................................................... 77
APPEDIX A ............................................................................................................................................. 79
6.14 Software Testing Certifications ............................................................................................ 79
REFERENCES .......................................................................................................................................... 80
3|PageNational Software Testing Guideline
TERMS AND DEFINITIONS
4|PageNational Software Testing Guideline
Actual Results Set of behaviours or conditions of a test item, or set of
conditions of associated data or the test environment,
observed as a result of test execution
EXAMPLE: outputs to hardware, changes to data, reports,
and communication messages sent
BSI British Standards Institution, the national standards body
of the United Kingdom
CEO Chief Executive Officer, the most senior executive with
responsibility for running an organization
CFO Chief Financial Officer, a senior executive with
responsibility for the financial affairs of an organization
Compliance A general concept of conforming to a rule, standard, law,
or requirement such that the assessment of compliance
results in a binomial result stated as "compliant" or
"noncompliant"
(A Guide to the Project Management Body of Knowledge
(PMBOK® Guide) -- Fifth Edition)
EXAMPLE: comply with a regulation.
Conformance Fulfilment by a product, process or service of specified
requirements
Defect Management A test practice involving the investigation and
documentation of test incident reports and the retesting
of defects when required
Dynamic Testing (1) Testing that requires the execution of the test item
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing--Part 1: Concepts and
definitions, 4.9)
(2) Testing that requires the execution of program code
(ISO/IEC/IEEE 29119-2:2013 Software and systems
engineering-- Software testing--Part 2: Test processes, 4.4)
Exhaustive Testing A testing approach where every possible combination of
5|PageNational Software Testing Guideline
input value is executed (infeasible for all practical test
items)
Expected Results Observable predicted behaviour of the test item under
specified conditions based on its specification or another
source
(ISO/IEC/IEEE 29119- 1:2013 Software and systems
engineering-- Software testing--Part 1: Concepts and
definitions, 4.15)
Experience-Based Testing A testing practice, in which testing is based on the tester's
previous experience, such as their domain knowledge and
knowledge of particular software and systems, along with
metrics from previous projects
Feature Set (1) Collection of items which contain the test conditions of
the test item to be tested which can be collected from
risks, requirements, functions, models, etc.
(ISO/IEC/IEEE 29119- 1:2013 Software and systems
engineering-- Software testing--Part 1: Concepts and
definitions)
(2) Logical subset of the test item(s) could be treated
independently of other feature sets in the subsequent test
design activities
(ISO/IEC/IEEE 29119-2:2013 Software and systems
engineering-- Software testing --Part 2: Test processes,
4.10)
ICT Information and Communication Technology
(ISO/IEC/IEEE 23026:2015 Systems and software
engineering--Engineering and management of websites for
systems, software, and services information, 5)
IEC The International Electrotechnical Commission is a global
organization for the preparation and publication of
International Standards for all electrical electronic and
6|PageNational Software Testing Guideline
related technologies.
IEEE The Institute of Electrical and Electronics Engineers in the
world's largest technical professional organization
dedicated to educational and technical advancement of
electrical and electronic engineering, telecommunications,
computer engineering and allied disciplines.
Incident Report Documentation of the occurrence, nature, and status of
an incident
(ISO/IEC/IEEE 29119- 1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.18)
Information Item Separate identifiable body of information that is
produced, stored, and delivered for human use
(ISO/IEC/IEEE 15289:2015 Systems and software
engineering--Content of life-cycle information products
(documentation). 5.11)
ISO The International Organization for Standardization (ISO) is
an international standard-setting body composed of
representatives from various national standards
organizations.
Lessons Learned The knowledge gained during a project which shows how
project events were addressed or should be addressed in
the future with the purpose of improving future
performance
(A Guide to the Project Management Body of Knowledge
(PMBOK ® Guide) -- Fifth Edition)
One-to-One Test Case An approach to test case design, where one test case is
Design derived for each test coverage item
Organizational Test A document that provides information about testing for
Specification an organization. i.e. information that is not project specific
(ISO/IEC/IEEE 29119-1:2013 Software and systems
7|PageNational Software Testing Guideline
engineering-- Software testing--Part 1: Concepts and
definitions, 4.24)
Organizational Test Strategy Documents that express the generic requirements for the
testing to be performed on all the projects run within the
organization, providing detail on how the testing is to be
performed
(ISO/IEC/IEEE 29119- 1:2013 Software and systems
engineering-- Software testing--Part 1: Concepts and
definitions, 4.25)
Product Risk Risk that a product could be defective in some specific
aspect of its function, quality, or structure
ISO/IEC/IEEE 29119-3:2013 Software and systems
engineering--Software testing-- Part 3: Test
documentation, 4.8)
Project Risk Risk related to the management of a project
ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing--Part 1: Concepts and
definitions, 4.31)
EXAMPLE: lack of staffing, strict deadlines, changing
requirements
Regression Testing Testing following modifications to a test item or to its
operational environment, to identify whether regression
failures occur
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing--Part 1: Concepts and
definitions, 4.32)
Requirements-Based Testing An approach to deriving test cases based on exercising
specified user requirements
8|PageNational Software Testing Guideline
Retesting Re-execution of test cases that previously returned a "fail"
result, to evaluate the effectiveness of interviewing
corrective actions
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing-- Part 1: Concepts and
definitions, 4.34)
SYN: confirmation testing
Risk Combination of the probability of an abnormal event or
failure and the consequence(s) of that event or failure to a
system's components, operators, users, or environment
(IEEE 1012-2012 IEEE Standard for System and Software
Verification and Validation, 3.1)
Risk Exposure Potential loss presented to an individual, project, or
organization by a risk
(ISO/IEC 16085:2006 Systems and software engineering--
Life cycle processes--Risk management, 3.10)
Risk Register A document in which the results of risk analysis and risk
response planning are recorded
(A Guide to the Project Management Body of Knowledge
(PMBOK® Guide) -- Fifth Edition)
NOTE: The risk register details all identified risks, including
description, category, cause, probability of occurring,
impact(s) on objectives, proposed responses, owners, and
current status. It can be kept in a database. See Also: risk
management plan
Risk-Based Testing Testing in which the management, selection, prioritization,
and use of testing activities and resources are consciously
based on corresponding types and levels of analyzed risk
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing--Part 1: Concepts and
definitions,4.35)
9|PageNational Software Testing Guideline
Scripted Testing (1) Dynamic testing in which the tester's actions are
prescribed by written instructions in a test case
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.37)
(2) Testing performed based on a documented test script
(ISO/IEC/IEEE 29119-2:2013 Software and systems
engineering--Software testing--Part 2: Test Processes,
4.23)
NOTE: Normally applies to manually executed testing,
rather than the execution of an automated script
Static Testing Testing in which a test item is examined against a set of
quality or other criteria without code being executed
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.42)
EXAMPLE: reviews, static analysis
SEE ALSO: inspection
Test Automation A test practice involving the automation of testing using
software that is usually referred to as test tools
NOTE: Automated testing is often considered to be mainly
concerned with the execution of scripted tests rather than
have the testers execute tests manually, however many
additional testing activities can be supported by software-
based tools.
Test Basis Body of knowledge used as the basis for the design of
tests and test cases
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering --Software testing--Part 1: Concepts and
definitions, 4.47)
NOTE: The test basis can take the form of documentation,
10 | P a g eNational Software Testing Guideline
such as a requirements specification, design specification,
or module specification, but can also be an undocumented
understanding of the required behaviour.
Test Case Set of test case preconditions, inputs (including actions,
where applicable), and expected results, developed to
drive the execution of a test item to meet test objectives,
including correct implementation, error identification,
checking quality, and other valued information
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering -- Software testing--Part 1: Concepts and
definitions, 4.48)
NOTE: A test case is the lowest level of test input (i.e. test
cases are not made up of test cases) for the test sub
process for which it is intended.
Test Completion Criteria A set of criteria to determine whether a specific test sub-
process can be considered as completed
EXAMPLE: 100% of statements exercised by test cases
EXAMPLE: Zero critical defects remain open
Test Completion Report A report that summarizes the testing that was performed
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.51)
SYN: test summary report
Test Condition Testable aspect of a component or system, such as a
function, transaction, feature, quality attribute, or
structural element identified as a basis for testing
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.52)
11 | P a g eNational Software Testing Guideline
Test Coverage Item An attribute or combination of attributes that is derived
from one or more test conditions by using a test design
technique that enables the measurement of the
thoroughness of the test execution
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.54)
Test Data Data created or selected to satisfy the input requirements
for executing one or more test cases, which can be
defined in the test plan, test case, or test procedure
(ISO/IEC/IEEE 29119-2:2013 Software and systems
engineering-- Software testing--Part 2: Test processes,
4.34)
Test Environment Facilities, hardware, software, firmware, procedures and
documentation intended for or used to perform testing of
software
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.60)
Test Execution A process of running a test on the test item, producing
actual results
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing--Part 1: Concepts and
definitions, 4.64)
Test Execution Log A document that records details of the execution of one or
more test procedures
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.65)
12 | P a g eNational Software Testing Guideline
Test Item Work product that is an object of testing
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.68)
EXAMPLE: A system, a software item, a requirements
document, a design specification, a user guide
SYN: test object
Test Level Specific instantiation of a test sub-process
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.69)
EXAMPLE: Component, component integration, system,
and acceptance testing
SYN: test phase
Test Management Planning, estimating, monitoring, reporting, control and
completion of test activities
(ISO/IEC/IEEE 29119-2:2013 Software and systems
engineering--Software testing--Part 2: Test processes,
4.49)
Test Measures Variable to which a value is assigned as the result of
measuring an attribute of the testing
EXAMPLE: 80% branch coverage
Test Objective Purpose of performing a testing activity associated with a
feature or feature set
EXAMPLE: Provision of information about product
qualities
Test Phase Specific instantiation of a test sub-process
(ISO/IEC/IEEE 29119-2:2013 Software and systems
engineering--Software testing--Part 2: Test Processes,
4.52)
SYN: test phase
13 | P a g eNational Software Testing Guideline
Test Plan Detailed description of test objectives to be achieved and
the means and schedule for achieving them, organized to
coordinate testing activities for some test item or set of
items
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
definitions, 4.75)
EXAMPLE: A project test plan (also known as a master test
plan) that encompasses all testing activities on the project;
further detail of particular test activities can be defined in
one or more test sub-process plans (i.e. a system test plan
or a performance test plan)
NOTE: It identifies test items, the features to be tested,
the testing tasks, who will do each task, and any risks
requiring contingency planning. Typical contents identify
the items to be tested, tasks to be performed,
responsibilities, schedules, and required resources for the
testing activity.
Test Policy Executive-level document that describes the purpose,
goals, principles and scope of testing within an
organization
(ISO/IEC/IEEE 29119-3:2013 Software and systems
engineering--Software testing--Part 3: Test
documentation, 4.26)
SYN: organizational test policy
Test Procedure Sequence of test cases in execution order, associated
actions to set up the initial preconditions, and wrap-up
activities post execution
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering--Software testing--Part 1: Concepts and
14 | P a g eNational Software Testing Guideline
definitions, 4.78)
Test Procedure Specification Document specifying one or more test procedures, which
are collections of test cases to be executed for a particular
objective
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing--Part 1: Concepts and
definitions, 4.79)
NOTE: A test procedure specification for an automated
test run is usually called a test script.
Test Process Process that provides information on the quality of a
software product
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing--Part 1: Concepts and
definitions. 4.80)
NOTE: Often comprised of a number of activities, grouped
into one or more test sub-processes
Test Result Indication of whether or not a specific test case has
passed or failed, i.e. if the actual result observed as test
item output corresponds to the expected result or if
deviations were observed
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing-- Part 1: Concepts and
definitions, 4.82)
Test Script Test procedure specification for manual or automated
testing
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing-- Part 1: Concepts and
definitions, 4.83)
15 | P a g eNational Software Testing Guideline
Test Set (1) Set of one or more test cases with a common
constraint on their execution
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing-- Part 1: Concepts and
definition, 4.84)
(2) Collection of test cases for the purpose of testing a
specific test objective
(ISO/IEC/IEEE 29119-2:2013 Software and systems
engineering-- Software testing-- Part 2: Test processes,
4.62)
NOTE: The test sets will typically reflect the feature sets,
but they could contain test cases for a number of feature
sets. Test cases for a test set could be selected based on
the identified risks, test basis, retesting, or regression
testing.
Test Status Report Report that provides information about the status of the
testing that is being performed in a specified reporting
period
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing-- Part 1: Concepts and
definitions, 4.86)
Test Strategy Part of the Test Plan that describes the approach to
testing for a specific test project or test sub-process or
sub-processes
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing-- Part 1: Concepts and
definitions, 4.87)
NOTE: The test strategy usually describes some or all of
the following: the test practices used; the test sub
processes to be implemented; the retesting and
regression testing to be employed; the test design
16 | P a g eNational Software Testing Guideline
techniques and corresponding test completion criteria to
be used; test data; test environment and testing tool
requirements; and expectations for test deliverables.
Test Sub-process Test management and dynamic (and static) test processes
used to perform a specific test level (e.g. system testing,
acceptance testing) or test type (e.g. usability testing,
performance testing) normally within the context of an
overall test process for a test project
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing-- Part 1: Concepts and
definitions, 4.88)
NOTE: Depending on the life cycle model used, test sub-
processes are also typically called test phases, test levels,
test stages or test tasks.
SYN: test sub process
Test Type Group of testing activities that are focused on specific
quality characteristics
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing-- Part 1: Concepts and
definitions, 4.91)
Example: security testing, functional testing, usability
testing, performance testing
Traceability Matrix Matrix that records the relationship between two or more
products of the development process
(ISO/IEC/IEEE 24765:2017 Systems and software
engineering-Vocabulary)
EXAMPLE: a matrix that records the relationship between
the requirements and the design of a given software
component
17 | P a g eNational Software Testing Guideline
Unscripted Testing Dynamic testing in which the tester's actions are not
prescribed by written instructions in a test case
(ISO/IEC/IEEE 29119-1:2013 Software and systems
engineering-- Software testing-- Part 1: Concepts and
definitions, 4.94)
Validation Confirmation, through the provision of objective evidence,
that the requirements for a specific intended use or
application have been fulfilled
(ISO/IEC 25000:2014 Systems and software Engineering--
Systems and software product Quality Requirements and
Evaluation (SQuaRE)-- Guide to SQuaRE, 4.41
Verification Confirmation, through the provision of objective evidence,
that specified requirements have been fulfilled
(ISO/IEC 25000:2014 Systems and software Engineering--
Systems and software product Quality Requirements and
Evaluation (SQuaRE)-- Guide to SQuaRE, 4.64
18 | P a g eNational Software Testing Guideline
EXECUTIVE SUMMARY
The importance of software testing cannot be over emphasised in today’s rapidly growing
technological environment; according to Forester publication (2018) the global tech market will grow
from 3.4% to 4% in 2017 and 2018 respectively; reaching a three (3) trillion dollar high in 2018. The
acquisition of technology driven solutions to drive growth in the public and private sectors have
continued to increase in Nigeria. The indigenous software market has not been left out of this growth
trend, but continues to suffer stiff competition from foreign off-the-shelf software used to meet local
needs, where indigenous software could have provided the appropriate solutions. According to
industry analysts, this trend has affected growth of the local software sector, which by now should be
in excess of probably ten (10) billion dollars annually if well harnessed. However, quality assurance
challenges amongst other factors have contributed to low adoption of indigenous software products
in the country.
In view of this, the National Information Technology Development Agency (NITDA) advocates a
structured approach to software quality assurance through a holistic software testing framework. The
National Software Testing Guideline (NSTG) adopts a multi-layer approach to software testing, which
is easily adaptable to any software domain to improve the quality of software and reduce cost of
development. Developed with the public and private sector in mind, the NSTG provides a practical
guide towards software testing as defined by the ISO/IEC/IEEE 29119 series of standards.
The guideline provides a brief introduction of NITDA’s purpose, its scope and objectives. It identifies
the relationship and importance of software testing in the Software Development Life Cycle (SDLC)
and introduces the multi-layer approach to testing, principles of software testing and the project
organisation structure.
The third section of the guideline focuses on the test approach, detailing the tree processes defined
by ISO/IEC/IEEE/29119; which are the Organisational Test Process, Test Management Processes and
Dynamic Test Process. It provides activities and tasks required during the test processes and practical
steps to implement software testing from planning to test execution.
Section four provides the important concept of testing levels, where every unit or component of
software is tested for defects and conformance with the intent of the system.
The fifth section provides details on testing techniques and its role in software testing. The final
section of the guideline provides details on test documentation and how it fits into the multi-layer
testing approach.
19 | P a g eNational Software Testing Guideline
1 INTRODUCTION
In a bid to create the necessary enabling environment for Information Technology (IT) for national
growth and development, the National Information Technology Development Agency (NITDA) is
focused on developing and regulating the IT sector in Nigeria, through the development of
appropriate guidelines and frameworks, to promote trust, efficiency, accuracy, reliability of
information systems in delivering information technology services. The National Software Testing
Guideline (NSTG) is developed on the premise that mitigating software vulnerability risks through the
promotion of structured software testing practice for safety and quality of software development will
create the enabling environment for growth of the indigenous testing sector in Nigeria.
The NSTG is developed based on international software testing standards which provides support for
a wide range of application areas in any software development life cycle; thus, being generic and can
be applied to different software domains.
1.1 Authority
In exercise of the mandate conferred on NITDA, specifically by section 6(b) and 6(e) of the National
Information Technology Agency Act of 2007; the National Software Testing Guideline is hereby issued.
1.2 Scope
This document shall play the specific role of ensuing a National Software Testing Guideline, developed
in line with international standard practices for software testing teams in initiating, planning,
managing, implementing and maintaining testing for software by public and private organizations;
and applicable to the testing of software developed, purchased or modified in-house.
1.3 Benefits of a National Software Testing Guideline
Every guide or standard brings about its own specific values and inherent benefits. This is obviously
the case with the NSTG, which stands to attain the following benefits among others:
Assist NITDA to achieve its slated mandate as required by Law, through effective
regulation of software practices in Nigeria;
It provides a standard guideline in identifying and fixing errors before the software
becomes operational, by reducing the risk of software failures considerably, which would
improve quality of software products developed in the country; thereby increasing user
satisfaction and demand for local software products;
Reduces cost of mitigation using structured risk-based techniques for testing and
maintaining software development and products;
20 | P a g eNational Software Testing Guideline
Improved reliability of software product by following set software testing technical
guidelines;
Improves integration of legacy and varying software platforms by providing assurance
that integration would work optimally with no performance retardation through
structured testing processes;
Helps to reduce production time spent on software integration for public and private
sectors; and,
Helps attain professional discipline in software testing practices in the country.
1.4 Objectives of The National Software Testing Guideline
The objectives of this document are:
To provide the software industry with a structured software testing guideline, which
offers best practices in software testing to promote safety and quality of software
development in Nigeria; and,
To provide an enabling environment for indigenous software testing administration,
management and sustainability, through structured testing guidelines to improve quality,
availability, reliability and reduce cost of software developed in Nigeria.
2 THE SOFTWARE DEVELOPMENT LIFECYCLE (SDLC)
The Software Development Life Cycle (SDLC) are phases in software development with an initiation
phase through to system maintenance. Each phase produces outputs required by the next phase in
the lifecycle. A typical SDLC is initiated by gathering business requirements, which are then analysed
and translated into a design; codes are then produced based on the design specification
(development phase). Software testing typically starts after coding is complete and before software is
deployed for use (deployment phase). During the deployment phase, the software is maintained
(maintenance/operation phase) until it is retired. There are many SDLC models used during software
development process, which include:
Waterfall model
Incremental model
V-model
Iterative model
RAD model
Agile model
21 | P a g eNational Software Testing Guideline
Spiral model
Prototype model
The NSTG is not developed upon any specific SDLC model but built in line with the ISO/IEC/IEEE 29119
standard which defines test terminology, processes, documentation and techniques that can be
adopted within the testing phase of any SDLC model. The following phases are predominately found
in all SDLC models:
Requirement gathering and analysis phase
Design phase
Implementation phase
Testing phase
Deployment phase
Maintenance Phase
SDLC STLC
REQUIREMENT & ANALYSIS PHASE REQUIREMENT SPECIFICATION
DESIGN PHASE TEST PLANNING
CODING PHASE TEST CASE DEVELOPMENT
TESTING PHASE TEST ENVIRONMENT SET-UP
DEPLOYMENT PHASE TEST EXECUTION
MAINTENNCE PHASE TEST CYCLE CLOSURE
Figure 1: SDLC and STLC Relationship Diagram
22 | P a g eNational Software Testing Guideline
The software testing phase starts after the codes have been developed, which are in turn tested
against the system requirement specification (test basis) gathered during the SDLC requirement
phase to make sure the software conforms with the intent of the development. During the testing
phase various tests are conducted to test for conformity, such tests are:
Functional testing;
Non-functional testing;
Unit testing;
Integration testing;
System testing; and,
Acceptance testing.
2.1 The Software Testing Life Cycle
Software developers follow the Software Development Life Cycle (SDLC), while software testers adopt
the Software Testing Life Cycle (STLC). The STLC refers to a testing process with distinct steps to be
executed in sequence to ensure that the quality goals of the software have been achieved. Each
phase of the STLC has different goals and deliverables, which provides a quality management process
for testing software. The STLC is initiated from the testing phase of the SDLC (See Figure 1: SDLC and
STLC Relationship Diagram) after coding is completed and again after deployment, where the
software is operational and being maintained until it’s retired.
The NSTG follows the STLC multi-layer testing process defined by ISO/IEC/IEEE 29119 standard, which
includes the Organisational Test Process, Test Management Process and the Dynamic Test Process.
ORGANISATION TEST PROCESS
TEST MANAGEMENT PROCESS
DYNAMIC TEST PROCESS
Figure 2: Multi-layer Approach Software Testing Process
23 | P a g eNational Software Testing Guideline
2.2 Software Testing Principles
While conducting software testing, the expectation is to achieve 100% test coverage within the
threshold of the testing process. To achieve this, the following testing principles are common
practice, which should be maintained:
1. Testing should be conducted through risk-based assessment and not on exhaustive
testing methods;
2. Testing should start early at the development phase of the SDLC, so that any defects in
the requirements and design phase are detected early. The earlier a defect is detected,
the cheaper it is to repair;
3. External test teams should be used for software testing implementation different from
those that developed the software;
4. All software testing processes must be comprehensive to test for invalid and unforeseen,
as well as valid and anticipated input conditions;
5. Any software testing specification must be detailed, complete and clear;
6. Test cases need to be regularly reviewed and revised, with new and different test cases to
find more defects;
7. If the software product does not meet the user’s requirements, there is no point in
testing to find defects and repairing it; and,
8. Do not assume in your test planning that no errors would be found in the system.
2.3 Project Organisation Structure
In IT projects, it is important to create a project organisation structure for the management of the
day-to-day activities of the project. In software testing projects, several organisational structures are
possible, each having its own inherent requirements depending on the use. For example, in safety
systems where strict adherence to safety is required, a Software Quality Assurance team would be
required in the project organisational structure as a subject specialist on quality of the system.
However, for this guideline, the Test Project Team is streamlined and composed of the Test Team
(Test Manager/Test Lead, Testers, Developer in Test and Test Administrator), Development Team
(Development Manager and Programmers) and Software Quality Assurance Members; and they all
report to the Project Manager.
The Project Manager in turn reports to the organisation’s management (Project Board). The test team
are external to the organisation and different from those who developed the software. This scenario
can also be applied to off-the-shelf purchased software and to external software development
24 | P a g eNational Software Testing Guideline
contractors. Depending on the nature of the software and the size of the organisation, the
organisation structure below can be adapted to fit any testing scenario as required.
PROJECT MANAGER
DEVELOPMENT MANAGER SQA TEST MANAGER
PROGRAMMERS TESTERS DEVELOPER IN TESTING TEST ADMINISTRATOR
Figure 3: Software Testing Project Organisation
Position Responsibilities
Project Manager (PM) Represents the client and manages the day to day activities on the
project level.
Test Manager (TM) TM manages the whole testing project processes:
Test Planning Process;
Test Management Process;
Test Execution Process; and,
defines the test project direction.
Tester Tester develop the Test Cases;
Generate the Test Procedures;
Executes the test;
Logs the results of the tests; and
Report the defect.
Developer in Test Creates the program to test the codes created by the developers
(unit and integration testing); and,
Creates the test automation scripts.
Test Administrator Builds the Test Environment;
25 | P a g eNational Software Testing Guideline
Manages and maintains the test environment assets;
Supports the test team to use the test environment for test
execution; and,
Decommissions and archives the test environment when testing
is completed.
Software Quality Assurance Provides quality assurance support on the test project.
2.4 Minimum Requirements to become a Software Test Professional
It is expected that a Software tester shall possess the following:
a. A bachelor’s degree in Computer Science, Information Systems, Software Engineering or a
related technical field;
b. Should be able to create and document automated and manual test plans and procedures,
execute tests, analyse results and report on test problems and anomalies (document bugs);
c. Perform software testing in all level of the design-develop-test-release-maintain software life
cycle;
d. Understand various development methodologies;
e. Possess thorough knowledge of several testing tools;
f. Be fluent in UNIX, Linux and/or Windows, as well as scripting and command-line tools;
g. Be an excellent communicator (written and verbal) with development, operations, product
management and customers;
h. Have working knowledge of various programming languages, such as Java, JavaScript, C# or
C++, SQL, Python, PHP and Ruby on Rails;
i. Have Professional Certification such as:
I. ISTQB Certified Tester: The American Software Testing Qualifications Board (ASTQB);
II. Certified Software Tester (CSTE): The International Software Certification Board
(ISCB);
j. 3 to 6 years of cognate experience depending on the certification level attained.
See the full list of Software testing certifications in Appendix A.
3 SOFTWARE TESTING GUIDELINE APPROACH
The NSTG adopts the multi-layer approach to software testing process (See Figure 4: Multi-Layer
Testing Process of ISO/IEC/IEEE 29119). This approach combines various standards that have been
sequentially integrated into a single structured standard for software testing. Noting that
26 | P a g eNational Software Testing Guideline
conformance to the standard is flexible and simple to adopt in any software testing domain. The
ISO/IEC/IEEE 29119 structure consist of the following standards:
BS 7925-1
ISO/IEC 33063
IEEE 829
ISO/IEC 20246
BS 7925-2
Organsational •Creation and Maintenance of Organisational Test
Policy, Strategy, Processes, Procedures and Other
Test Process Assets.
Test Management •Test Planning Process;
•Test Monitoring and Control Process; and,
Processes •Test Completion Process.
•Test Design and Implementation Process;
Dynamic Test •Test Environment Set-up and Maintenance Process;
Processes •Test Execution Process; and,
•Test Incident Reporting Process.
Figure 4: Multi-Layer Testing Process
3.1 Organisational Test Process
This process is adopted for the development and management of the Organisational Test
Specifications (OTS). The Organisation Test Process is instantiated twice to develop the Organisational
Test Policy and Organisational Test Strategy. The OTS generally applies to testing throughout the
organisation.
3.1.1 Organisational Test Specifications
The following documents are components of the Organisational Test Specifications:
a. Organisational Test Policy: Executive level document that lays out the purpose, goals and
scope of testing in the organisation. It also provides the mechanism for establishment,
review and updates of the Organisation’s Test Policy, Test Strategy and approach to
project test management (See Test Management Process in section 3.3).
27 | P a g eNational Software Testing Guideline
b. Organisational Test Strategy: A detailed technical document, defining how testing is
carried out in the organisation. It’s a generic document (programme level), which
provides guidelines for software testing projects in the organisation.
NOTES:
The OTS is generic (programme oriented) and not specific to a test project. E.g. testing for
different test levels and test types may require different approaches which may not be defined in
the Organisational Test Strategy but defined in the project Test Strategy during test planning by
the Test Manager.
ORGANISATION TEST ORGANISATION TEST PROCESS
PROCESS (TEST POLICY)
ORGANISATION TEST PROCESS
(TEST STRATEGY)
Organisation Test Process initiated twice, once
each to create and maintain the Organisational
Test Policy and Organisational Test Strategy
Figure 5: Organisation Test Specification Process
3.1.1.1 Purpose
The purpose of the Organisation Test Process is to develop, monitor conformance and maintain
Organisational Test Specifications.
3.1.1.2 Outcomes
a. Requirements for the Organisational Test Specifications are identified;
b. Organisation Test Specifications document is developed;
28 | P a g eNational Software Testing Guideline
c. Stakeholder buy-in is solicited for the Organisational Test Specifications;
d. The Organisational Test Specifications is made accessible across the organisation;
e. Conformance to the OTS is monitored;
f. Stakeholders agree to updates to the OTS; and,
g. Updates are affected to the OTS document.
3.1.1.3 Activities and tasks
The following activities and tasks are to be implemented in line with the applicable organisational
policies and procedures by the department (e.g. ICT department) or person (Director/Head of ICT)
responsible for the Organisational Test Specifications:
a. Develop the Organisational Test Specifications through gathering requirements from
existing; testing practices, stakeholders, documents, workshops, interviews and other
relevant resources;
b. The developed requirements are used to create the Organisational Test Specifications
document;
c. Stakeholders approval is required on the content of the Organisational Test
Specifications; and,
d. The Organisational Test Specifications availability is communicated to relevant
stakeholders.
3.1.2 Practical Guideline
The following steps should be followed while maintaining the OTS.
3.1.2.1 Monitor and Control Use of Organisational Test Specifications
a. Monitor usage of OTS to determine if conformance is maintained; and,
b. Keep stakeholders aligned with the Organisational Test Specifications document
updates.
3.1.2.2 Update Organisational Test Specifications
a. Review all practice feedback of the Organisational Test Specifications;
b. Effectiveness of use and management of the Organisational Test Specifications should
be considered and feedback on changes to improve its effectiveness should be
determined and approved by stakeholders;
c. Implement all improvement changes that have been approved; and,
d. All changes to the document should be communicated to all stakeholders.
29 | P a g eNational Software Testing Guideline
NOTES:
Stakeholders in this process refers to the organisation’s management (Directors, CEO, CTO,
Heads of ICT, etc) and internal Project Managers’ involved in testing projects.
Find Organisational Test Policy and Organisational Test Strategy document template in Test
Documentation section.
3.2 Test Management Processes
This process covers the management of testing for a test project (project test management), and test
management for test levels (unit test, integration test, system test, acceptance test) and test types
(usability, accessibility, performance, configuration, security, availability tests etc) within a test
project.
Test Management Process initiated
once at the project level
(i.e. small test projects)
TEST MANAGEMENT TEST MANAGEMENT PROCESS
PROCESS (PROJECT LEVEL)
TEST MANAGEMENT PROCESS
TEST MANAGEMENT PROCESS
TEST MANAGEMENT PROCESS
(TEST LEVELS & TEST TYPES )
Test Management Process several instances
initiated at the test levels & for test types
(i.e. large test projects)
Figure 6: Test Management Process
The Test Management Process includes:
a. Test Planning;
b. Test Monitoring and control; and,
30 | P a g eYou can also read
