I. PRAD
The
Product Requirement Analysis Document is the document prepared/reviewed by
marketing, sales, and technical product managers. This document defines the
requirements for the product, the "What". It is used by the developer
to build his/her functional specification and used by QA as a reference for the
first draft of the Test Strategy.
II. Functional Specification
The
functional specification is the "How" of the product. The functional
specification identifies how new features will be implemented. This document
includes items such as what database tables a particular search will query.
This document is critical to QA because it is used to build the Test Plan.
QA is
often involved in reviewing the functional specification for clarity and
helping to define the business rules.
III. Test Strategy
The Test
Strategy is the first document QA should prepare for any project. This is a
living document that should be maintained/updated throughout the project. The first
draft should be completed upon approval of the PRAD and sent to the developer
and technical product manager for review.
The Test
Strategy is a high-level document that details the approach QA will follow in
testing the given product. This document can vary based on the project, but all
strategies should include the following criteria:
- Project Overview - What is the project.
- Project Scope - What are the core
components of the product to be tested
- Testing - This section defines the test
methodology to be used, the types of testing to be executed (GUI, Functional,
etc.), how testing will be prioritized, testing that will and will not be done
and the associated risks. This section should also outline the system
configurations that will be tested and the tester assignments for the project.
- Completion Criteria - These are the
objective criteria upon which the team will decide the product is ready for
release
- Schedule - This should define the
schedule for the project and include completion dates for the PRAD, Functional
Spec, and Test Strategy etc. The schedule section should include build delivery
dates, release dates and the dates for the Readiness Review, QA Process Review,
and Release Board Meetings.
- Materials Consulted - Identify the documents
used to prepare the test strategy
- Test Setup - This section should
identify all hardware/software, personnel pre-requisites for testing. This
section should also identify any areas that will not be tested (such as 3rd
party application compatibility.)
IV. Test Matrix (Test Plan)
The Test
Matrix is the Excel template that identifies the test types (GUI, Functional
etc.), the test suites within each type, and the test categories to be tested.
This matrix also prioritizes test categories and provides reporting on test
coverage.
- Test Summary report
- Test Suite Risk Coverage report
Upon
completion of the functional specification and test strategy, QA begins
building the master test matrix. This is a living document and can change over
the course of the project as testers create new test categories or remove
non-relevant areas. Ideally, a master matrix need only be adjusted to include
near feature areas or enhancements from release to release on a given product
line.
V. Test Cases
As
testers build the Master Matrix, they also build their individual test cases.
These are the specific functions testers must verify within each test category
to qualify the feature. A test case is identified by ID number and prioritized.
Each test case has the following criteria:
- Purpose - Reason for the test case
- Steps - A logical sequence of steps the
tester must follow to execute the test case
- Expected Results - The expected result
of the test case
- Actual Result - What actually happened
when the test case was executed
- Status - Identifies whether the test
case was passed, failed, blocked or skipped.
- Pass - Actual result matched expected
result
- Failed - Bug discovered that represents
a failure of the feature
- Blocked - Tester could not execute the
test case because of bug
- Skipped - Test case was not executed
this round
- Bug ID - If the test case was failed,
identify the bug number of the resulting bug.
VI. Test Results by Build
Once QA
begins testing, it is incumbent upon them to provide results on a consistent
basis to developers and the technical product manager. This is done in two
ways: A completed Test Matrix for each build and a Results Summary document.
For each
test cycle, testers should fill in a copy of the project's Master Matrix. This
will create the associated Test Coverage reports automatically (Test Coverage
by Type and Test Coverage by Risk/Priority). This should be posted in a place
that necessary individuals can access the information.
Since the
full Matrix is large and not easily read, it is also recommended that you
create a short Results Summary that highlights key information. A Results
Summary should include the following:
- Build Number
- Database Version Number
- Install Paths (If applicable)
- Testers
- Scheduled Build Delivery Date
- Actual Build Delivery Date
- Test Start Date
- Scope - What type of testing was planned
for this build? For example, was it a partial build? A full-regression build?
Scope should identify areas tested and areas not tested.
- Issues - This section should identify
any problems that hampered testing, represent a trend toward a specific problem
area, or are causing the project to slip. For example, in this section you
would note if the build was delivered late and why and what its impact was on
testing.
- Statistics - In this section, you can
note things such as number of bugs found during the cycle, number of bugs
closed during the cycle etc.
VII. Release Package
The
Release Package is the final document QA prepares. This is the compilation of
all previous documents and a release recommendation. Each release package will
vary by team and project, but they should all include the following
information:
- Project Overview - This is a synopsis of
the project, its scope, any problems encountered during the testing cycle and
QA's recommendation to release or not release. The overview should be a
"response" to the test strategy and note areas where the strategy was
successful, areas where the strategy had to be revised etc.
The
project overview is also the place for QA to call out any suggestions for
process improvements in the next project cycle.
Think of
the Test Strategy and the Project Overview as "Project bookends".
- Project PRAD - This is the Product
Requirements Analysis Document, which defines what functionality was approved
for inclusion in the project. If there was no PRAD for the project, it should
be clearly noted in the Project Overview. The consequences of an absent PRAD
should also be noted.
- Functional Specification - The document
that defines how functionality will be implemented. If there was no functional
specification, it should be clearly noted in the Project Overview. The
consequences of an absent Functional Specification should also be noted.
- Test Strategy - The document outlining
QA's process for testing the application.
- Results Summaries - The results
summaries identify the results of each round of testing. These should be
accompanied in the Release Package by the corresponding reports for Test
Coverage by Test Type and Test Coverage by Risk Type/Priority from the
corresponding completed Test Matrix for each build. In addition, it is
recommended that you include the full Test Matrix results from the test cycle
designated as Full Regression.
- Known Issues Document - This document is
primarily for Technical Support. This document identifies workarounds, issues
development is aware of but has chosen not to correct, and potential problem
areas for clients.
- Installation Instruction - If your
product must be installed as the client site, it is recommended to include the
Installation Guide and any related documentation as part of the release
package.
- Open Defects - The list of defects
remaining in the defect tracking system with a status of Open. Technical
Support has access to the system, so a report noting the defect ID, the problem
area, and title should be sufficient.
- Deferred Defects - The list of defects
remaining in the defect tracking system with a status of deferred. Deferred
means the technical product manager has decided not to address the issue with
the current release.
- Pending Defects - The list of defects
remaining in the defect tracking system with a status of pending. Pending
refers to any defect waiting on a decision from a technical product manager
before a developer addresses the problem.
- Fixed Defects - The list of defects
waiting for verification by QA.
- Closed Defects - The list of defects
verified as fixed by QA during the project cycle.
The
Release Package is compiled in anticipation of the Readiness Review meeting. It
is reviewed by the QA Process Manager during the QA Process Review Meeting and
is provided to the Release Board and Technical Support.
- Readiness Review Meeting:
The
Readiness Review meeting is a team meeting between the technical product
manager, project developers and QA. This is the meeting in which the team
assesses the readiness of the product for release.
This
meeting should occur prior to the delivery of the Gold Candidate build. The
exact timing will vary by team and project, but the discussion must be held far
enough in advance of the scheduled release date so that there is sufficient
time to warn executive management of a potential delay in the release.
The
technical product manager or lead QA may schedule this meeting.
- QA Process Review Meeting:
The QA
Process Review Meeting is meeting between the QA Process Manager and the QA
staff on the given project. The intent of this meeting is to review how well or
not well process was followed during the project cycle.
This is
the opportunity for QA to discuss any problems encountered during the cycle
that impacted their ability to test effectively. This is also the opportunity
to review the process as whole and discuss areas for improvement.
After
this meeting, the QA Process Manager will give a recommendation as to whether
enough of the process was followed to ensure a quality product and thus allow a
release.
This
meeting should take place after the Readiness Review meeting. It should be
scheduled by the lead QA on the project.
- Release Board Meeting:
This
meeting is for the technical product manager and senior executives to discuss
the status of the product and the teams release recommendations. If the results
of the Readiness meeting and QA Process Review meeting are positive, this
meeting may be waived.
The
technical product manager is responsible for scheduling this meeting.
This
meeting is the final check before a product is released.
Due to
rapid product development cycles, it is rare that QA receives completed PRADs
and Functional Specifications before they begin working on the Test Strategy,
Test Matrix, and Test Cases. This work is usually done in parallel.
Testers
may begin working on the Test Strategy based on partial PRADs or confirmation
from the technical product manager as to what is expected to be in the next
release. This is usually enough to draft out a high -level strategy outlining
immediate resource needs, potential problem areas, and a tentative schedule.
The Test
Strategy is then updated once the PRAD is approved, and again when the
functional specifications are complete enough to provide management with a
committed schedule. All drafts of the test strategy should be provided to the
technical product manager and it is QA's responsibility to ensure that
information provided in the document (such as potential resource problems) is
clearly understood.
If the
anticipated release does not represent a new product line, testers can begin
the Master Test Matrix and test cases at the same time the project's PRAD is
being finalized. Testers can build and/or refine test cases for the new
functionality as the functional specification is defined. Testers often
contribute to and are expected to be involved in reviewing the functional
specification.
The
results summary document should be prepared at the end of each test cycle and
distributed to developers and the technical product manager. It is designed
more to inform interested parties on the status of testing and possible impact
to the overall project cycle.
