Beautiful Testing: Leading Professionals Reveal How They Improve Software

by Tim Riley (Editor), Adam Goucher (Editor)

Overview

Successful software depends as much on scrupulous testing as it does on solid architecture or elegant code. But testing is not a routine process, it's a constant exploration of methods and an evolution of good ideas.

Beautiful Testing offers 23 essays from 27 leading testers and developers that illustrate the qualities and techniques that make testing an art. Through personal anecdotes, you'll learn how each of these professionals developed beautiful ways of testing a wide range of products — valuable knowledge that you can apply to your own projects.

Here's a sample of what you'll find inside:

• Microsoft's Alan Page knows a lot about large-scale test automation, and shares some of his secrets on how to make it beautiful
• Scott Barber explains why performance testing needs to be a collaborative process, rather than simply an exercise in measuring speed
• Karen Johnson describes how her professional experience intersected her personal life while testing medical software
• Rex Black reveals how satisfying stakeholders for 25 years is a beautiful thing
• Mathematician John D. Cook applies a classic definition of beauty, based on complexity and unity, to testing random number generators

All author royalties will be donated to the Nothing But Nets campaign to save lives by preventing malaria, a disease that kills millions of children in Africa each year.

This book includes contributions from:

• Adam Goucher
• Linda Wilkinson
• Rex Black
• Martin Schröder
• Clint Talbert
• Scott Barber
• Kamran Khan
• Emily Chen
• Brian Nitz
• Remko Tronçon
• Alan Page
• Neal Norwitz
• Michelle Levesque
• Jeffrey Yasskin
• John D. Cook
• Murali Nandigama
• Karen N. Johnson
• Chris McMahon
• Jennitta Andrea
• Lisa Crispin
• Matt Heusser
• Andreas Zeller
• David Schuler
• Tomasz Kojm
• Adam Christian
• Tim Riley
• Isaac Clerencia

Product Details

ISBN-13:	9780596159818
Publisher:	O'Reilly Media, Incorporated
Publication date:	11/06/2009
Series:	Theory in Practice
Pages:	329
Sales rank:	963,048
Product dimensions:	6.90(w) x 9.00(h) x 0.90(d)

About the Author

Tim Riley is the Director of Quality Assurance at Mozilla. He has tested software for 18 years including everything from spacecraft simulators, ground control systems, high security operating systems, language platforms, application servers, hosted services and open source web applications. He has managed software testing teams in startups to large corporations consisting of 3 to 120 people in size and in up to 6 countries. He has a software patent for a testing execution framework which matches test suites to available test systems. He enjoys being a breeder caretaker for Canine Companions for Independence (cci.org) along with live and studio sound engineering.

Adam Goucher has been testing software professionally for over ten years. In that time he has worked with start-ups, large multi-nationals and ones in between in both traditional and agile testing environments. A believer in the communication of ideas big and small, he writes frequently at http://adam.goucher.ca and teaches testing skills at a Toronto area technical college. In his off hours he can be found either playing or coaching box lacrosse - and then promptly applying lessons learned to testing. He is also an active member of the Association for Software Testing.

Table of Contents

Preface; How This Book Is Organized; Using Code Examples; Safari® Books Online; How to Contact Us; Acknowledgments; Beautiful Testers; Chapter 1: Was It Good for You?; Chapter 2: Beautiful Testing Satisfies Stakeholders; 2.1 For Whom Do We Test?; 2.2 What Satisfies?; 2.3 What Beauty Is External?; 2.4 What Beauty Is Internal?; 2.5 Conclusions; Chapter 3: Building Open Source QA Communities; 3.1 Communication; 3.2 Volunteers; 3.3 Coordination; 3.4 Events; 3.5 Conclusions; Chapter 4: Collaboration Is the Cornerstone of Beautiful Performance Testing; 4.1 Setting the Stage; 4.2 100%?!? Fail; 4.3 The Memory Leak That Wasn’t; 4.4 Can’t Handle the Load? Change the UI; 4.5 It Can’t Be the Network; 4.6 Wrap-Up; Beautiful Process; Chapter 5: Just Peachy: Making Office Software More Reliable with Fuzz Testing; 5.1 User Expectations; 5.2 What Is Fuzzing?; 5.3 Why Fuzz Test?; 5.4 Fuzz Testing; 5.5 Future Considerations; Chapter 6: Bug Management and Test Case Effectiveness; 6.1 Bug Management; 6.2 The First Step in Managing a Defect Is Defining It; 6.3 Test Case Effectiveness; 6.4 Case Study of the OpenSolaris Desktop Team; 6.5 Conclusions; 6.6 Acknowledgments; 6.7 References; Chapter 7: Beautiful XMPP Testing; 7.1 Introduction; 7.2 XMPP 101; 7.3 Testing XMPP Protocols; 7.4 Unit Testing Simple Request-Response Protocols; 7.5 Unit Testing Multistage Protocols; 7.6 Testing Session Initialization; 7.7 Automated Interoperability Testing; 7.8 Diamond in the Rough: Testing XML Validity; 7.9 Conclusions; 7.10 References; Chapter 8: Beautiful Large-Scale Test Automation; 8.1 Before We Start; 8.2 What Is Large-Scale Test Automation?; 8.3 The First Steps; 8.4 Automated Tests and Test Case Management; 8.5 The Automated Test Lab; 8.6 Test Distribution; 8.7 Failure Analysis; 8.8 Reporting; 8.9 Putting It All Together; Chapter 9: Beautiful Is Better Than Ugly; 9.1 The Value of Stability; 9.2 Ensuring Correctness; 9.3 Conclusions; Chapter 10: Testing a Random Number Generator; 10.1 What Makes Random Number Generators Subtle to Test?; 10.2 Uniform Random Number Generators; 10.3 Nonuniform Random Number Generators; 10.4 A Progression of Tests; 10.5 Conclusions; Chapter 11: Change-Centric Testing; 11.1 How to Set Up the Document-Driven, Change-Centric Testing Framework?; 11.2 Change-Centric Testing for Complex Code Development Models; 11.3 What Have We Learned So Far?; 11.4 Conclusions; Chapter 12: Software in Use; 12.1 A Connection to My Work; 12.2 From the Inside; 12.3 Adding Different Perspectives; 12.4 Exploratory, Ad-Hoc, and Scripted Testing; 12.5 Multiuser Testing; 12.6 The Science Lab; 12.7 Simulating Real Use; 12.8 Testing in the Regulated World; 12.9 At the End; Chapter 13: Software Development Is a Creative Process; 13.1 Agile Development As Performance; 13.2 Practice, Rehearse, Perform; 13.3 Evaluating the Ineffable; 13.4 Two Critical Tools; 13.5 Software Testing Movements; 13.6 The Beauty of Agile Testing; 13.7 QA Is Not Evil; 13.8 Beauty Is the Nature of This Work; 13.9 References; Chapter 14: Test-Driven Development: Driving New Standards of Beauty; 14.1 Beauty As Proportion and Balance; 14.2 Agile: A New Proportion and Balance; 14.3 Test-Driven Development; 14.4 Examples Versus Tests; 14.5 Readable Examples; 14.6 Permanent Requirement Artifacts; 14.7 Testable Designs; 14.8 Tool Support; 14.9 Team Collaboration; 14.10 Experience the Beauty of TDD; 14.11 References; Chapter 15: Beautiful Testing As the Cornerstone of Business Success; 15.1 The Whole-Team Approach; 15.2 Automating Tests; 15.3 Driving Development with Tests; 15.4 Delivering Value; 15.5 A Success Story; 15.6 Post Script; Chapter 16: Peeling the Glass Onion at Socialtext; 16.1 It’s Not Business…It’s Personal; 16.2 Tester Remains On-Stage; Enter Beauty, Stage Right; 16.3 Come Walk with Me, The Best Is Yet to Be; 16.4 Automated Testing Isn’t; 16.5 Into Socialtext; 16.6 A Balanced Breakfast Approach; 16.7 Regression and Process Improvement; 16.8 The Last Pieces of the Puzzle; 16.9 Acknowledgments; Chapter 17: Beautiful Testing Is Efficient Testing; 17.1 SLIME; 17.2 Scripting; 17.3 Discovering Developer Notes; 17.4 Oracles and Test Data Generation; 17.5 Mindmaps; 17.6 Efficiency Achieved; Beautiful Tools; Chapter 18: Seeding Bugs to Find Bugs: Beautiful Mutation Testing; 18.1 Assessing Test Suite Quality; 18.2 Watching the Watchmen; 18.3 An AspectJ Example; 18.4 Equivalent Mutants; 18.5 Focusing on Impact; 18.6 The Javalanche Framework; 18.7 Odds and Ends; 18.8 Acknowledgments; 18.9 References; Chapter 19: Reference Testing As Beautiful Testing; 19.1 Reference Test Structure; 19.2 Reference Test Extensibility; 19.3 Building Community; Chapter 20: Clam Anti-Virus: Testing Open Source with Open Tools; 20.1 The Clam Anti-Virus Project; 20.2 Testing Methods; 20.3 Summary; 20.4 Credits; Chapter 21: Web Application Testing with Windmill; 21.1 Introduction; 21.2 Overview; 21.3 Writing Tests; 21.4 The Project; 21.5 Comparison; 21.6 Conclusions; 21.7 References; Chapter 22: Testing One Million Web Pages; 22.1 In the Beginning…; 22.2 The Tools Merge and Evolve; 22.3 The Nitty-Gritty; 22.4 Summary; 22.5 Acknowledgments; Chapter 23: Testing Network Services in Multimachine Scenarios; 23.1 The Need for an Advanced Testing Tool in eBox; 23.2 Development of ANSTE to Improve the eBox QA Process; 23.3 How eBox Uses ANSTE; 23.4 How Other Projects Can Benefit from ANSTE; Contributors; Colophon;