Preface
Acknowledgments
Part I What Is Automated Testing?
1 The Birth and Evolution of Automated Testing
1.1 Automated Testing
1.2 Background on Software Testing
1.3 The Automated Test Life-Cycle Methodology(ATLM)
1.3.1 Decision to Automate Test
1.3.2 Test Tool Acquisition
1.3.3 Automated Testing Introduction Phase
1.3.4 Test Planning,Design,and Development
1.3.5 Execution and Management of Tests
1.3.6 Test Program Review and Assessment
1.4 ATLM's Role in the SoftwareTesting Universe
1.4.1 ATLM Relationship to System Development Life Cycle
1.4.2 Test Maturity Model(TMM)-Augmented by Automated Software Testing Maturity
1.4.3 Test Automation Development
1.4.4 Test Effort
1.5 Software Testing Careers
2 Decision to Automate Test
2.1 Overcoming False Expectations for Automated Testing
2.1.1 Automatic Test Plan Generation
2.1.2 Test Tool Fits All
2.1.3 Immediate Test Effort Reduction
2.1.4 Immediate Schedule Reduction
2.1.5 Tool Ease of Use
2.1.6 Universal Application of Test Automation
2.1.7 One Hundred Percent Test Coverage
2.2 Benefits of Automated Testing
2.2.1 Production of a Reliable System
2.2.2 Improvement of the Quality of the Test Effort
2.2.3 Reduction of Test Effort and Minimization of Schedule
Case Study:Value of Test Automation Measurement
2.3 Acquiring Management Support
2.3.1 Test Tool Proposal
3 Automated Test Tool Evaluation and Selection
3.1 The Organization's Systems Engineering Environment
3.1.1 Third-Party Input from Management,Staff,and Customers and End Users
3.1.2 Tool Criteria Reflecting the Systems engineering Environment
3.1.3 Level of Software Quality
3.1.4 Help Desk Problem Reports
3.1.5 Budget Constraints
3.1.6 Types of Tests
3.1.7 Long-Term Investment Considerations
3.1.8 Test Tool Process
3.1.9 Avoiding Shortcuts
3.2 Tools That Support the Testing Life Cycle
3.2.1 Business Analysis Phase Tools
3.2.2 Requirements Definition Phase Tools
3.2.3 Tools for the Analysis and Design Phase
3.2.4 Programming Phase Tools
3.2.5 Metrics Tools
3.2.6 Other Testing Life-Cycle Support Tools
3.2.7 Testing Phase Tools
3.3 Test Tool Research
3.3.1 Improvement Opportunities
3.4 Evaluation Domain Definition
3.5 Hands-On Tool Evaluation
3.5.1 Evaluation Report
3.5.2 License Agreement
Part II Introduction of Automated Testing to a Project
4 Automated Testing Introduction Process
4.1 Test Process Analysis
4.1.1 Process Review
4.1.2 Goals and Objectives of Testing
Case Study:Test Objectives and Strategies
4.1.3 Test Strategies
4.2 Test Tool Consideration
4.2.1 Review of Project-Specific System Requirements
4.2.2 Application-Under-Test Overview
4.2.3 Review of Project Schedule
4.2.4 Test Tool Compatibility Check
4.2.5 Demonstration of the Tool to the Project Team
4.2.6 Test Tool Support Profile
4.2.7 Review of Training Requirements
5 Test Team Management
5.1 Organizational Structure of a Test Team
5.1.1 Stovepipe Test Team
5.1.2 Centralized Test Team
5.1.3 IV&V Test Team
5.1.4 Systems Methodology and Test Team
5.1.5 Test Team Summary
5.2 Test Program Tasks
5.3 Test Effort Sizing
5.3.1 Test Team Sizing Methods:An Overview
5.3.2 Development Ratio Method
5.3.3 Percentage Method
5.3.4 Test Procedure Method
5.3.5 Task Planning Method
5.3.6 Test Effort Sizing Factors
5.4 Test Engineer Recruiting
5.4.1 Test Engineer Qualities
5.4.2 Test Team Composition
5.4.3 Job Requisition
5.4.4 Recruiting Activities
5.4.5 Locating Test Engineers
5.4.6 Test Engineer Interviews
5.4.7 Distinguishing the Best Candidate
5.5 Roles and Responsibilities
Part III Test Planning and Preparation
6 Test Planning:Smart Application of Testing
6.1 Test Planning:Smart Application of Testing
6.2 Test Planning Activities
6.2.1 System Description
6.2.2 Critical/High-Risk Functions
6.2.3 Test Goals,Objectives,and Strategies
6.2.4 Test Tools
6.2.5 Test Program Parameters
6.2.6 Verification Methods
6.2.7 Teset Requirements Definition
6.3 Test Requirements Management
6.3.1 Requirement Management Tools
6.3.2 Assessing the Test Requirements Risk
6.3.3 Prioritization of Tests
6.3.4 Requirements Traceability Matrix
6.4 Test Program Events,Activities,and Documentation
6.4.1 Events
6.4.2 Activities
6.4.3 Documentation
6.5 The Test Environment
6.5.1 Test Environment Preparations
6.5.2 Test Environment Integration and Setup
6.6 The Test Plan
6.6.1 Test Completion/Acceptance Criteria
6.6.2 Sample Test Plan
7 Test Analysis and Design
7.1 Test Requirements Analysis
7.1.1 Development-Level Test Analysis(Structural Approach)
7.1.2 System-Level Test Analysis(Behavioral Approach)
7.2 Test Program Design
7.2.1 Test Program Design Models
7.2.2 White-Box Techniques(Development-Level Tests)
7.2.3 Black-Box Techniques(System-Level Tests)
7.2.4 Test Design Documentation
7.3 Test Procedure Design
7.3.1 Test Procedure Design
7.3.2 Test Procedure Definition
7.3.3 Automated Versus Manual Test Analysis
Case Study:Naming Conventions
7.3.4 Manual Test Design Guidelines
7.3.5 Detailed Test Design
7.3.6 Test Data Requirements
8 Test Development
8.1 Test Development Architecture
8.1.1 Technical Environment
8.1.2 Environment Readiness Checks
8.1.3 Automation Framework Reuse Analysis
8.1.4 Test Procedure Development/Execution Schedule
8.1.5 Modularity-Relationship Analysis
8.1.6 Explanation of the Sample Modularity-Felationship Matrix
8.1.7 Calibration of the Test Tool
8.1.8 Compatibility Work-Around Solutions
Case Study:Incompatibility Work-Around Solutions
8.1.9 Manual Execution of Test Procedures
8.1.10 Test Procedure Inspections-Peer Reviews
8.1.11 Test Procedure Configuration Management
8.2 Test Development Guidelines
8.2.1 Design-to-Development Transition
8.2.2 Reusable Test Procedures
Case Study:Navigation Using Tabs or Mouse Clicks
Case Study:Testing Bitmaps Using a Capture/Playback Tool
8.2.3 Maintainable Test Procedures
Case Study:Automating Documentation
Case Study:Automated Random Testing
8.2.4 Other Guidelines
8.3 Automation Infrastructure
8.3.1 Table-Driven Test Automation
8.3.2 PC Environment Automated Setup Script
8.3.3 Automated Recording Options
8.3.4 Login Function
8.3.5 Exit Function
8.3.6 Navigation
8.3.7 Verifying GUI Standards
8.3.8 Smoke Test
Case Study:Smoke Test Application
8.3.9 Error-Logging Routine
8.3.10 Help Function Verification Script
8.3.11 Timed Message Boxes Function
8.3.12 Advanced Math Functions
Part IV Test Execution and Review
9 Test Execution
9.1 Executing and Evaluating Test Phases
9.1.1 Unit Test Execution and Evaluation
9.1.2 Integration Test Execution and Evaluation
9.1.3 System Test Execution and Evaluation
9.1.4 Test Results Analysis of Regression Tests
9.1.5 User Acceptance Test Execution and Evaluation
9.2 Defect Tracking and New Build Process
9.2.1 Defect Life-Cycle Model
9.3 Test Program Status Tracking
9.3.1 Earned Value Management System
Case Study:System Test Status Tracking
9.3.2 Test Metrics Collection and Analysis
10 Test Program Review and Assessment
10.1 Test Program Lessons Learned-Corrective Actions and Improvement Activity
10.2 Test Program Return on Investment
Case Study:Test Program Return on Investment
Case Study:Quantify Tool Return on Investment
Part V Appendixes
A How to Test Requirements
A.1 Requirements Testing Approach
Abstract
The Quality Gateway
Make the Requirement Measurable
Quantifiable Requirements
Nonquantifiable Requirements
Keeping Track
Coherency and Consistency
Completeness
Relevance
Requirement or Solution?
Stakeholder Value
Traceaility
Order in a Disorderly World
Conclusions
References
B Tools That Support the Automated Testing Life Cycle
B.1 Introduction
B.2 Business Analysis Phase
B.2.1 Business Modeling Tools
B.2.2 Configuration Management Tools
B.2.3 Defect Tracking Tools
B.2.4 Technical Review Management
B.2.5 Documentation Generators
B.3 Requirements Definition Phase
B.3.1 Requirements Management Tools
B.3.2 Requirements Verifiers
B.3.3 Use Case Generators
B.4 Analysis and Design Phase
B.4.1 Visual Modeling Tools
B.4.2 Structure Charts,Flowcharts,and Sequence Diagrams
B.4.3 Test Procedure Generators
B.5 Programming Phase
B.5.1 Syntax Checkers/Debuggers
B.5.2 Memory Leak and Runtime Error Detection Tools
B.5.3 Code Checkers
B.5.4 Static and Dynamic Analyzers
B.5.5 Unit and Integration Test Tools
B.6 Metrics Tools
B.6.1 Code(Test)Coverage Analyzers and Code Instrumentors
B.6.2 Usability Measurement Tools
B.7 Testing Support Tools
B.7.1 Test Data Generators
B.7.2 File Comparison Tools
B.7.3 Simulation Tools
B.8 Testing Phase
B.8.1 Test Management Tools
B.8.2 Network Testing Tools
B.8.3 GUI Application Testing Tools
B.8.4 Load/Performance Testing Tools
B.8.5 Web Testing Tools
B.8.6 Year 2000 Testing Tools
B.9 Other Test Tool Vendors
C Test Engineer Development
C.1 Technical Skills Stage
C.2 Test Process Stage
C.3 Team Effort Stage
C.4 Technical Stewardship Stage
C.5 Test/Project Management Stage
C.6 Business/Product Management Stage
D Sample Test Plan
D.1 Introduction
D.1.1 Purpose
D.1.2 Background
D.1.3 System Overview
D.1.4 Applicable Documents
D.1.5 Master Schedule
D.2 Roles and Responsibilities
D.2.1 Project Organization
D.2.2 Project Roles and Responsibilities
D.2.3 Test Task Structure
D.2.4 Test Team Resources
D.3 Test Program
D.3.1 Scope
D.3.2 Test Approach
D.3.3 Test Strategies
D.3.4 Automated Tools
D.3.5 Qualification Methods
D.3.6 Test Requirements
D.3.7 Test Design
D.3.8 Test Development
D.4 Test environment
D.4.1 Test Environment Configuration
D.4.2 Test Data
D.5 Test Execution
D.5.1 Test Program Reporting
D.5.2 Test Program Metrics
D.5.3 Defect Tracking
D.5.4 Configuration Management
D.6 Detailed Test Schedule
APPENDIXES
D.A Test Procedure Development guidelines
D.B Test Verification Summary and Matrix
D.C Test Procedures and Test Scripts
E Best Practices
E.1 Documented Proess
E.2 Managing Expectations
E.3 Pilot Project
E.4 Test Tool Compatibility Checks
E.5 Test Tool Upgrades
E.6 Baselined System Setup and Configuration
E.7 Software Installations in the Test Environment Baseline
E.8 Overall Test Program Objectives
E.9 Keep Automation Simple
E.10 Test Procedure Design and Development Standards
E.11 Automated Versus Manual Test Analysis
E.12 Reuse Analysis
E.13 Test Team Communication with Other Teams
E.14 Schedule Compatibility
E.15 Customer Involvement
E.16 Defect Documentation and Reporting
E.17 Automated Test Advocates and Experts
E.18 Test Team Assignments
E.19 User Group Participation
E.20 Test Tool Improvement Suggestions
E.21 Become a Beta Testing Site
E.22 Specialty Topic Experts
Index
鄂公网安备 42010302001612号 