The ROI of Testing
Shaun Bradshaw, Questcon Technologies

In today’s competitive business environment, corporations need and demand a good return on investment (ROI) for everything they do—and testing is no exception. Although executive managers are requesting meaningful metrics more often than ever, many test managers are struggling to justify the cost versus benefit of their departments’ work. Often these test managers are unsure how to calculate investment costs versus dollars saved when using solid QA and testing methodologies. Test managers need the business tools and techniques to provide this business critical information, not only to satisfy upper management but also to ensure their departments are indeed making a positive contribution. Shaun Bradshaw demonstrates the tangible benefits of testing on the software development lifecycle by defining ROI in the context of software testing and defect prevention. In addition, Shaun provides several key metrics you can use to demonstrate the quantitative benefits of testing.

• Prevention (QA) costs versus detection (testing) costs
• Key metrics for calculating testing’s value to the business
• Testing ROI calculation examples

Monty Python's Flying Test Lab
Robert Sabourin, AmiBug.com

And now for something completely different . . . Monty Python's Flying Circus revolutionized comedy and brought zany British humor to a world-wide audience. However, buried deep in the hilarity and camouflaged in its twisted wit, lie many important testing lessons—tips and techniques you can apply to real world problems to deal with turbulent projects, changing requirements, and stubborn project stakeholders. Rob Sabourin examines some of the most famous Python bits—“The Spanish Inquisition” telling us to expect the unexpected, “The Dead Parrot” asking if we should really deliver this product to the customer, “The Argument” teaching us about bug advocacy, “Self Defense Against Fresh Fruit” demonstrating the need to pick the right testing tool, and a host of other goofy gags, each one with a lesson for testers.

• How to test effectively with persistence
• Make your point with effective communication
• Keys ways to clarify project goals and requirements

For Success, Build Record/Playback into Your Application
Gerard Meszaros, ClearStream Consulting

Stories about failed attempts to automate functional testing are very easy to find and have given the record/playback style test automation a black eye. Is this approach fundamentally flawed or can the business benefits of automated testing be realized through recorded tests? The flaw with most commercial record/playback tools is that they are intended for use with existing applications that have not been designed for testability. Therefore, the tools can only interact with the application through the user interface, making execution slow and prone to flakiness because user interfaces make terrible machine interfaces. Gerard Meszaros introduces the concept of designing testability and test recording capabilities directly into the application. This approach allows automated tests to interact with the application via a programming API to make them much more robust. Case studies illustrate how to design record/playback test automation into different application technologies and how this technique impacts the application design. 

• Why general purpose record/ playback tools have fatal flaws
• How recording can work well when built onto the program
• Designing testability into software design to improve your application

Performance Testing in Enterprise Application Environments
David Chadwick, IBM

As systems become more complex—serving the enterprise and implemented on the Web and across the Internet—performance testing is becoming more important and more difficult. David Chadwick suggests that the starting point is to design tests that reflect real user activity, including independent arrivals of transactions and varying input data to prevent “cache only” results. David explains how to break down the end-to-end system response time into the distributed components involved in processing the transactions. Learn to use resource-monitoring data to discover bottlenecks on individual systems. By examining the frequency and time spent in various processes, performance testers can determine where resources are being consumed and how to tune a system for better performance. Gain a new understanding of specific bottlenecks: repeated calls for individual items that could be returned as a set, requesting unfiltered data and filtering it repeatedly, inefficient structures and algorithms for sorting and searching, and lack of proper local caching.

• Recording and generating Web testing sessions
• Challenges of testing modern Web systems
• Using response time data to find software inefficiencies

Lightning Talks: A Potpourri of 5-Minute Presentations
Facilitated by Dawn Haynes

Lightning Talks are nine five-minute talks in one conference session. Lightning Talks represent a much smaller investment of time than track speaking and offer the chance to try conference speaking without the heavy commitment. Lightning Talks are an opportunity to present your single biggest bang-for-the-buck idea quickly. Use this as an opportunity to give a first time talk or to present a new topic for the first time. Maybe you just want to ask a question, invite people to help you with your project, boast about something you did, or tell a short cautionary story. These things are all interesting and worth talking about, but there might not be enough to say about them to fill up a full conference session.

Seven Habits of Highly Effective Automation Testers
Krishna Iyer & Mukesh Mulchandani, ZenTEST Labs

In many organizations, test automation is becoming a specialized career path. Mukesh Mulchandani identifies seven habits of highly effective automation specialists and compares them with Stephen Covey’s classic Seven Habits of Highly Effective People. Mukesh not only describes behavior patterns of effective automation testers but he also discusses how to internalize these patterns so that you use them instinctively. Drawing on his experience of managing large test automation projects for financial applications, Mukesh describes obvious habits such as saving and reusing tests. He then describes the uncommon but essential habits of strategizing, seeking, selling, and communicating. Learn how to avoid the bad habits that automation test novices—and even experts—may unconsciously adopt.

• Keys to successful test automation  
• Leadership skills needed by test automation specialists
• List of automation problem behaviors

Avoid Performance Testing Data Deception
Ben Simo, Standard and Poors

Don't be fooled by your performance test results. Performance testing can easily generate an unwieldy amount of data—some relevant and some not. Testers and their tools often use statistical methods to make sense of the data, but using statistics requires sacrificing accuracy and thoroughness. The good news is that we do not need to understand all the details to make good use of test results. The challenge is to determine what information really matters and how to present it in a useful manner. Join Ben Simo as he addresses  common performance test statistical problems including built-in bias, agreeable averages, invisible inadequacies, gargantuan groupings, stingy sets, mountainous molehills, creative charting, alien alliances, and more. Find out how statistical reporting can deceive rather than inform—often unintentionally—and recognize what the numbers do not say. By learning how to tell the truth with performance test results, you can give your stakeholders the information they really need to make good decisions. 

• What is takes to understand performance data
• How to protect yourself from misinformation
• Provide useful and understandable performance information to stakeholders

Optimize Your Testing with Virtual Test Lab Automation
Brad Johnson, Borland Software

The complex nature of software development often requires testing on multiple hardware platforms, operating systems, Web and application servers, and databases. Add to it the many different builds, patches, and regionalized versions that development delivers and you understand the immense challenge faced by test engineers trying to provide adequate test coverage. Adding virtual lab automation to your testing process can help your organization overcome these challenges and may dramatically improve the way you test—at a fraction of the cost of traditional multi-system approaches. Brad Johnson explores ways to seamlessly integrate virtual environments into the software testing process and explains how virtual test labs enable a test team to test more efficiently, across a wider range of environments, and with greater coverage of critical requirements. Join Brad to see if virtual test lab automation can improve software quality across your organization. 

• Leverage virtualization technologies to create a more efficient test environment
• Seamlessly integrate virtual environments into testing
• Test across a wider range of environments for increased coverage

Test Strategies for the Modern Distributed World
Dan Koloski, Empirix

Enterprise application development is quickly evolving with SOA and Web 2.0 taking center stage. Organizational structures are changing, with growing numbers of testing teams employing offshore resources. What do these changes mean to you, and what should you do to prepare? Most testing groups were created based on traditional development processes, traditional application architectures, and traditional organizational structures. As agile enters the mainstream, more change is on the way. Outsourcing, offshore development, and acquisitions continuously change the organizational landscape. Dan Koloski discusses proven and practical practices for adapting to today’s new technologies, new structures, and the new modern distributed world. He will discuss how to effectively communicate across virtual and physical silos as well as ways to adapt your test strategies and execution to component-based applications. Finally, you will learn how your organization can leverage Open Source applications and the Software-as-a-Service (SaaS) delivery model.

• Why traditional test groups are organized the way they are
• New technologies and new organizations for new testing approaches
• Ways to leverage new technologies to your advantage

Telling Your Exploratory Story
Jon Bach, Quardev

What do you say when your manager asks you, “How did it go today?” If you have a pile of test cases on your desk, it may be acceptable for you to say, “I ran x% of these tests today,” or “I’ll be finished with this stack in y days at the rate I’m going.”  However, if you’re using exploratory testing as your approach, it may be downright terrifying to try to give a status report, especially if project stakeholders think exploratory testing is irresponsible and downright reckless compared to pre-scripted test cases. So how, then, can you retain the freedom and power of exploration, and yet give a report that earns you credibility, respect, and perhaps more autonomy? Jon Bach suggests ways to describe and explain the critical and creative thinking that drives your exploratory testing so that others have a better chance of understanding and appreciating your value as an exploratory tester.

• Sixteen skills and tactics for exploratory testing
• Strategies for taking useful notes during exploration
• Preparing for stakeholder scrutiny of exploratory testing

Client-Side Attacks: The New Vulnerability
Matt Fisher, HP

Historically, we have focused on server-side security vulnerabilities rather than their client-side counterparts. As cybercrime continues to evolve, the sophistication of client-side attacks is increasing and the severity of these vulnerabilities is growing. The advent of phishing and efforts to create botnet armies have exploded in recent years due to their profit potential. Client-side issues such as vulnerabilities in Web browsers and file corruption have become the facilitators, which make these attacks possible.  Matt Fisher demonstrates examples in which client-side vulnerabilities have been leveraged for criminal gain. Matt walks through typical attack scenarios to help you better understand how these attacks succeed—and how you can combat them. He'll also peer into his crystal ball in an attempt to anticipate how such attacks will evolve in the future.
 

• Vulnerabilities in Web browsers and how to combat them
• Hacking client systems for criminal gain
• Phishing scams and botnet armies on the attack

Pairwise Testing Comes of Age
George Sherwood, Testcover.com

You’ve heard of orthogonal arrays and pairwise testing. Perhaps you’ve used a pairwise test case generator tool. Have you ever wondered where these popular and powerful techniques originated? Actually they have been around for almost twenty years. During this time, important test design principles have emerged and choices for test generation tools have improved. George Sherwood, inventor of CATS, one of the first pairwise test tools, reviews what we have learned and how it applies to testing today. He shows the benefits of using pairwise test techniques for selecting configurations and for generating test data. George also outlines important considerations for successful pairwise test designs, including the problems to anticipate and avoid. George dispels the mystery of pairwise test generators with a simplified view of how they work. He explains how recent advances in computing platforms and mathematical constructions will improve the next generation of tools.
 

• Lessons learned from years of pairwise testing experience
• Simple pairwise test tactics for normal test cases and error conditions
• What is beyond orthogonal arrays for more complex testing challenges

Five Steps to Becoming a Top Tester
Bernie Berger, Liquidnet Holdings, Inc.

Have you ever wondered what top testers do that enables them to accomplish so much more than an average tester? As a tester, the key to maximizing your potential for success is taking responsibility for developing your own testing skills. Bernie Berger shares five testing tips that can help you strengthen your testing ability and set you apart from the crowd. Learn what it means to know your stakeholders; why it is important to think concurrently; where to look for boundaries in testing your application; how to test beyond the specifications; and when to be skeptical. The common theme of these tips is: If you want to improve the way you test, you first have to improve the way you think. Bernie believes that testing is more than just a job—it is a cool way to earn a living, and we can all do it better.

• The hidden stakeholders in your system under test
• Testing at the boundary +1 and -1 may not be sufficient
• How to construct complex test scenarios using current events

Test Estimation: Simple Models and Practical Lessons
Tonnvane Wiswell, Total Jobs Group

As software testers, we are regularly asked how long it will take to test a system. Unfortunately, we rarely have the tools to produce an accurate estimate. Tonnvane Wiswell introduces methods for producing better estimates—best guess, experienced person's best guess, and ways to use past data as a baseline—and the advantages and disadvantages of each. She discusses adaptable formulas that incorporate “buffer time” and risk factors. Finally, Tonnvane presents a real life example of a testing project with solid time estimates, including an explanation of how team size was determined, how the work flow was designed, what the “actual hours” of testing were, what unexpected items affected the testing time, and how the project permanently changed the company's “test estimation formula.”  

• Two simple estimations methods when you don’t have historical data
• How to improve test estimation in your organization
• Test estimation examples from the real world

The Promise of Model-Based Testing
Mahesh Velliyur, Maveric Testing Solutions

Good test design is the cornerstone of every test effort. Although approximately 50% of testing time is spent on test design, very little has been done to bring structure, automation, and a scientific approach to the test design process itself. The quality of designs is highly dependent on the individual tester’s expertise. Mahesh Velliyur N has found that model-based testing is extremely useful in testing end-to-end business scenarios. Model-based testing brings a rigor to test design by focusing on what and how much to test. Models are useful in testing only when they are able to generate a useful set of test cases providing measurable coverage and traceability back to the system under test. Often, model-based testing meets those requirements. In addition, it avoids duplication of tests at different test levels, ensures more complete data coverage, and helps in prioritizing test execution. Mahesh demonstrates a simple functional model he developed for retail banking and the algorithms developed  in test case design and sequencing.  
 

• The fundamentals of model-based testing
• Domain-specific frameworks using model-based testing principles
• Practical examples of model-based testing

Finding Backdoor Threats with Static Analysis
Chris Wysopal, Veracode

According to research from Gartner, 75% of all new security attacks are against applications and 90% of all vulnerabilities reside within software. However, enterprise IT security continues to be concentrated on the network to protect the perimeter from external attack rather than detecting vulnerabilities on the inside. In some of the world’s largest businesses, there’s evidence that malicious users may be deliberately leaving “backdoor” vulnerabilities to be exploited later when applications are put into full production. Methods are available to detect backdoors in your software, with static analysis as the most effective technique available. Chris Wysopal explains the technology and benefits of binary and source code static analysis and presents various techniques for inspecting software for backdoors along with the pros and cons of each method. Chris shows examples of how backdoors present themselves in software and how best to find them.
 

• Why backdoors in software are difficult to detect
• Manual and automated detection of backdoors
• How static binary analysis works

Practical Pairwise Testing with PICT
BJ Rollison, Microsoft

Fault analysis reveals that interaction between the variables of dependent parameters is a common source of failure in complex systems. Imagine you are assigned to test a feature with twenty independent parameters and five possible states for each parameter. The total number of possible combinations is greater than five-hundred billion. At one test executed per millisecond, it would take more than 3,000 years to test all possible combinations. So, which combinations do we test? Pairwise testing is a systematic procedure to reduce the total number of tests by selecting a set of tests that evaluates every pair, rather than every combination. BJ Rollison compares the orthogonal arrays method to pairwise analysis and provides a detailed example of how to use PICT, a powerful, highly configurable combinatorial analysis tool that is freely available. Learn to systematically test complex interdependent parameters and improve your test coverage with fewer tests.  
 

• Advantages of pairwise testing over orthogonal arrays
• A model file for smart pairwise analysis
• Ways to measure the effectiveness of pairwise testing

Today's Testing Innovations 
Lee Copeland, Software Quality Engineering

As a consultant, Lee Copeland has spoken with thousands of software testers in hundreds of different organizations. Generally, he comes away from these discussions depressed with the state of testing. Many organizations neither know about nor have adopted recent important innovations in our field. Lee will discuss nine of the important innovations in testing—the context-driven school, testing specialties, test-first development, really good books, open source tools, session-based test management, testing workshops, certification, and freedom of the press. Join Lee for his list, and propose others if you’d like. Discuss the keys to innovation and take a test evaluating your organization’s innovation quota.
 

• The top nine innovations in testing today, plus the ones you add
• How to be more innovative in your testing
• Evaluate your organization's openness to innovation