Reading Notes of Acceptance Test Engineering Guide

The Acceptance Test Engineering Guide will provide guidance for technology stakeholders (developers, development leads, testers, test leads, architects, etc.) and business stakeholders (managers, customers, end users, etc) on the discipline of acceptance testing.

What is acceptance testing?

Acceptance Testing: Planned evaluation of a system by customers/customer proxies to assess to what degree it satisfies their expectations.

Reading Notes

• Test automation is software development.
• Much of the cost of software development is maintenance - changing the software after it is written.
• Successful organizations understand that test automation is software development, that it involves significant maintenance costs, and that they can and must make deliberate, vigilant effort to keep maintenance costs low.
• The need to change tests comes from two directions: changes in requirements and changes in the system’s implementation.
• If we can’t stop requirements and implementations from changing, the only way to keep the maintenance cost of tests low is to make the tests adaptable to those kinds of changes.
• Developers have learned - often through painful experience - that two key factors make code difficult to change: Incidental details and duplication.
• An acceptance test investigates a system to determine whether it correctly implements a given responsibility.
• The essence of an acceptance test is the responsibility it investigates, regardless of the technology used to implement the test.
• Duplication destroys maintainability.
• If you write automated tests so that they express system responsibilities clearly and directly, and if you remove duplication, you will significantly reduce the maintenance costs that arise from both changes in requirements and changes in system implementation.
• One of the key forms of waste in software development is handoffs between highly-specialized roles.
• Bug Triage which divided the bugs into three categories with respect to the next milestone or release: Must Fix, Would Like to Fix (if we have time), and Will Not Fix.
• Triage should only look at new bugs - re-examing old bugs repeatedly is waste.
• The most effective way to ensure that poor quality software gets delivered is to ask the supplier to over commit.
• Time-boxed incremental development is a proven technique for achieving on-time delivery of quality software.
• A feature is an independently testable unit of functionality that is either directly visible to a customer (customer-facing feature) or is a piece of infrastructure that will be consumed by another feature.
• Building quality in - start with the end in mind.
• The ultimate goal of any development process is to build the right product of high quality and to build it with minimum waste.
• Defect Prevention before Defect Detection.
• Automated unit and component tests define the design intent of the software.
• In lean thinking such as exemplified by the lean manufacturing paradigm used by Toyota, unfinished work (inventory) is considered a form of waste. In software development, software that has been written but has not been accepted is unfinished work.
• Functional tests should be run as soon as the corresponding functionality is built. This should include business workflow tests that verify end to end business processes, use case tests that verify the various scenarios of a single use case or business transaction, and business rule tests that ensure that business rules are implemented correctly.
• Business-Facing Tests : When we (Product Owner, often together with someone from the Product Development Team) prepare these tests before development starts, we can be sure that the Product Development Team understands what they need to build. This is known as Acceptance Test-Driven Development.
• Technology-Facing Tests: The Test-Driven Development process has been shown to significantly improve the quality of the software in several ways including better software structure, reduced software complexity, and fewer defects found during acceptance testing.
• Test Strategy, one of the key decisions is what kinds of tests should be automated and which approach to testing should be used for manual test. The goal of these decisions is to try to minimize project risk while also minimizing the time and effort spent testing the software.
• Exploratory testing is an effective and efficient approach to testing that leverages the intelligence of the testers to maximize the bugs found in a fixed amount of time.
• A common strategy on projects that have an extensive suite of automated tests is to run these tests first as a form of regression test as the first activity in a test cycle; this is a form of extended smoke test. This ensures that the software functions properly before a human tester spends any time doing manual testing.
• Developers may prepare unit tests to verify that the code they wrote has successfully achieved the design intent.
• System testing is done at the level of the whole product and can include a range of system sizes: individual systems or integrated workflows through multiple systems that have to interact.
• Component tests are used for testing packages, individual executables or subsystems.
• The granularity of unit testing can be on little things like methods / functions or bigger things like modules and classes.
• An alternative to using a plan-driven approach within the test cycle is to use a more interative style known as Session-Based Test Management. Session-Based Test Management is commonly used with exploratory testing.
• Exploratory testing is summarized as "Simultaneous test design and execution with an emphasis on learning".

Related Readings

• Acceptance Test Engineering Guide
• Unit Testing with NSubstitute
• Building the Testing Pipeline