USF Libraries
USF Digital Collections

Understanding roi metrics for software test automation

MISSING IMAGE

Material Information

Title:
Understanding roi metrics for software test automation
Physical Description:
Book
Language:
English
Creator:
Jayachandran, Naveen
Publisher:
University of South Florida
Place of Publication:
Tampa, Fla.
Publication Date:

Subjects

Subjects / Keywords:
Return on investment metrics
Automated testing feasibility
Manual vs. automated testing
Software quality assurance
Regression testing
Repetitive testing
Dissertations, Academic -- Computer Science -- Masters -- USF   ( lcsh )
Genre:
government publication (state, provincial, terriorial, dependent)   ( marcgt )
bibliography   ( marcgt )
theses   ( marcgt )
non-fiction   ( marcgt )

Notes

Abstract:
ABSTRACT: Software test automation is widely accepted as an efficient software testing technique. However, automation has failed to deliver the expected productivity more often than not. The goal of this research was to find out the reason for these failures by collecting and understanding the metrics that affect software test automation and provide recommendations on how to successfully adopt automation with a positive return on investment (ROI). The metrics of concern were schedule, cost and effectiveness. The research employed an experimental study where subjects worked on individual manual and automated testing projects. The data collected were cross verified and supplemented with additional data from a feedback survey at the end of the experiment. The results of this study suggest that automation involves a heavy initial investment in terms of schedule and cost, which needs to be amortized over subsequent test cycles or even subsequent test projects.
Thesis:
Thesis (M.S.)--University of South Florida, 2005.
Bibliography:
Includes bibliographical references.
System Details:
System requirements: World Wide Web browser and PDF reader.
System Details:
Mode of access: World Wide Web.
Statement of Responsibility:
by Naveen Jayachandran.
General Note:
Title from PDF of title page.
General Note:
Document formatted into pages; contains 99 pages.
General Note:
Includes vita.

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
aleph - 001670373
oclc - 62310805
usfldc doi - E14-SFE0001241
usfldc handle - e14.1241
System ID:
SFS0025562:00001


This item is only available as the following downloads:


Full Text
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam Ka
controlfield tag 001 001670373
003 fts
005 20051216093329.0
006 m||||e|||d||||||||
007 cr mnu|||uuuuu
008 051121s2005 flu sbm s000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0001241
035
(OCoLC)62310805
SFE0001241
040
FHM
c FHM
049
FHMM
090
QA76 (Online)
1 100
Jayachandran, Naveen.
0 245
Understanding roi metrics for software test automation
h [electronic resource] /
by Naveen Jayachandran.
260
[Tampa, Fla.] :
b University of South Florida,
2005.
502
Thesis (M.S.)--University of South Florida, 2005.
504
Includes bibliographical references.
516
Text (Electronic thesis) in PDF format.
538
System requirements: World Wide Web browser and PDF reader.
Mode of access: World Wide Web.
500
Title from PDF of title page.
Document formatted into pages; contains 99 pages.
Includes vita.
3 520
ABSTRACT: Software test automation is widely accepted as an efficient software testing technique. However, automation has failed to deliver the expected productivity more often than not. The goal of this research was to find out the reason for these failures by collecting and understanding the metrics that affect software test automation and provide recommendations on how to successfully adopt automation with a positive return on investment (ROI). The metrics of concern were schedule, cost and effectiveness. The research employed an experimental study where subjects worked on individual manual and automated testing projects. The data collected were cross verified and supplemented with additional data from a feedback survey at the end of the experiment. The results of this study suggest that automation involves a heavy initial investment in terms of schedule and cost, which needs to be amortized over subsequent test cycles or even subsequent test projects.
590
Adviser: Dewey Rundus.
Co-adviser: Alan Hevner
653
Return on investment metrics.
Automated testing feasibility.
Manual vs. automated testing.
Software quality assurance.
Regression testing.
Repetitive testing.
690
Dissertations, Academic
z USF
x Computer Science
Masters.
773
t USF Electronic Theses and Dissertations.
4 856
u http://digital.lib.usf.edu/?e14.1241



PAGE 1

Understanding ROI Metrics for Software Test Automation by Naveen Jayachandran A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science Department of Computer Science and Engineering College of Engineering University of South Florida Co-Major Professor: Dewey Rundus, Ph.D. Co-Major Professor: Alan Hevner, Ph.D. Member: Rafael Perez, Ph.D. Date of Approval: June 2, 2005 Keywords: Return on investment metrics, automated testing feasibility, manua l vs. automated testing, software quality assurance, regression testing, repetiti ve testing Copyright 2005, Naveen Jayachandran

PAGE 2

Dedication To my Parents, Suja, Rocky and Reemo.

PAGE 3

Acknowledgements I would like to thank my major professors Dr. Alan Hevner and Dr. Dewey Rundus for recognizing and encouraging my interest, giving me this opportunity and providing constant support for my research. I have gained immensely from this experience enabling me to truly appreciate the saying – ‘The journey is more important than the destination’. I would like to acknowledge my faculty committee member, Dr. Rafael Perez for his time and valuable comments. This work was supported in part by the National Institute for Systems Test and Productivity (NISTP) at the Unive rsity of South Florida under the US Space and Naval Warfare Systems Command Contract No. N00039-02-C-3244. I would like to thank Tom Barr, Judy Hyde and the rest of the NISTP team for their useful insights and thoughts. I also acknowledge IBM R ational for providing the Rational Test Suite that has been used extensively in this research st udy. Finally, I thank my parents and my sister for being my source of motivation and hope.

PAGE 4

i Table of Contents List of Tables.....................................................................................................................iv List of Figures.....................................................................................................................v Abstract..............................................................................................................................vi Chapter 1: Introduction.......................................................................................................1 1.1 Prelude................................................................................................................1 1.2 The Problem........................................................................................................2 1.3 The Goals............................................................................................................2 1.4 The Questions.....................................................................................................3 1.5 The Metrics.........................................................................................................3 1.6 The Experiment...................................................................................................4 Chapter 2: Literature Review..............................................................................................5 2.1 Software Testing.................................................................................................5 2.1.1 Manual Testing............................................................................................5 2.1.2 Automated Testing.......................................................................................6 2.2 Metrics................................................................................................................6 2.2.1 Metrics for Software Testing.......................................................................6 2.3 Return on Investment (ROI)...............................................................................7 2.4 Related Work......................................................................................................8

PAGE 5

ii 2.4.1 COCOMO II................................................................................................8 2.4.2 Rational Suite Economic Value Model........................................................9 Chapter 3: Experimental Design.......................................................................................10 3.1 The Experiment.................................................................................................10 3.1.1 Print Dialog Testing (P1)...........................................................................12 3.1.2 File Dialog Testing (P2).............................................................................12 3.2 The Metrics.......................................................................................................13 3.3 Data Collection.................................................................................................14 3.4 Other Details.....................................................................................................15 Chapter 4: Results & Discussion......................................................................................16 4.1 Demographic Data............................................................................................16 4.2 Schedule............................................................................................................18 4.2.1 Regression Testing.....................................................................................21 4.2.2 Pure Repetitive Testing..............................................................................24 4.3 Cost...................................................................................................................31 4.3.1 Fixed Costs.................................................................................................31 4.3.2 Variable Costs............................................................................................32 4.3.3 Regression Testing.....................................................................................35 4.3.4 Pure Repetitive Testing..............................................................................35 4.3.5 Accounting for Fixed Costs.......................................................................35 4.4 Effectiveness.....................................................................................................37 4.4.1 Lessons Learned.........................................................................................41 4.5 Other Metrics....................................................................................................41

PAGE 6

iii 4.5.1 Scope of Testing........................................................................................42 4.5.2 Depth of Testing........................................................................................43 4.5.3 Test Tool Adoption....................................................................................45 4.5.4 User Ratings for Effectiveness & Productivity..........................................48 4.5.5 Documentation and Traceability................................................................49 4.5.6 Automation – Individual Vs. Group Effort................................................50 4.5.7 Test Coverage............................................................................................50 4.6 Tripod SDLC Model.........................................................................................51 4.6.1 Automation Requirements.........................................................................52 4.6.2 Automation Feasibility...............................................................................53 4.6.3 Framework Design.....................................................................................53 4.7 Contributions.....................................................................................................53 4.8 Limitations........................................................................................................54 4.8.1 Project Scope.............................................................................................54 4.8.2 Error Seeding.............................................................................................55 4.8.3 Automation Tool........................................................................................55 Chapter 5: Conclusion.......................................................................................................56 5.1 Future Research................................................................................................57 References.........................................................................................................................59 Appendices........................................................................................................................62

PAGE 7

iv List of Tables Table 1: Experimental Setup (G – Group, P – Project, M – Manual, A – Automated)....12 Table 2: Project Timeline..................................................................................................14 Table 3: 1 st Cycle Time Periods – Manual Vs. Automated..............................................21 Table 4: Regression Testing: 2 nd Cycle Time Periods – Manual Vs. Automated.............22 Table 5: Regression Testing: 3 rd Cycle Time Periods – Manual Vs. Automated.............23 Table 6: Regression Testing: Automated Testing Cycle Schedule Overflo ws................24 Table 7: Pure Repetitive Testing: 2 nd Cycle Time Periods – Manual Vs. Automated......25 Table 8: Pure Repetitive Testing: 3 rd Cycle Time Periods – Manual Vs. Automated......26 Table 9: Pure Repetitive Testing: Automated Testing Cycle Schedule Overf lows..........27 Table 10: Regression Testing: Testing Cycle Variable Cost Overflows..........................35 Table 11: Pure Repetitive Testing: Testing Cycle Variable Cost Overflow s...................35 Table 12: Actual Defects & False Positives – Manual Vs. Automated............................38 Table 13: ROI Recommendations.....................................................................................56

PAGE 8

v List of Figures Figure 1: Experimental Setup...........................................................................................11 Figure 2: Subject’s Programming and Testing Experience..............................................17 Figure 3: 1 st Cycle Time lines – Manual Vs. Automated.................................................21 Figure 4: Regression Testing: 2 nd Cycle Time lines – Manual Vs. Automated................22 Figure 5: Regression Testing: 3 rd Cycle Time lines – Manual Vs. Automated................23 Figure 6: Pure Repetitive Testing: 2 nd Cycle Time Lines – Manual Vs. Automated.......25 Figure 7: Pure Repetitive Testing: 3 rd Cycle Time Lines – Manual Vs. Automated........26 Figure 8: Timeline Comparison – Manual Vs. Regression Vs. Pure Repetitive..............28 Figure 9: First Cycle Variable Cost – Manual Vs. Automated.........................................33 Figure 10: Actual Defects & False Positives – Manual Vs. Automated...........................37 Figure 11: Effect of Programming Rating........................................................................45 Figure 12: Scripting Experience.......................................................................................46 Figure 13: Training Effectiveness Rating.........................................................................47 Figure 14: Test Tool Usability Rating..............................................................................47 Figure 15: Effectiveness & Productivity Rating...............................................................48 Figure 16: Documentation Effort......................................................................................49 Figure 17: Tripod SDLC Model.......................................................................................52

PAGE 9

vi Understanding ROI metrics for Software Test Automation Naveen Jayachandran ABSTRACT Software test automation is widely accepted as an efficient software t esting technique. However, automation has failed to deliver the expected productivity more often than not. The goal of this research was to find out the reason for these failures by collecting and understanding the metrics that affect software test autom ation and provide recommendations on how to successfully adopt automation with a positive return on investment (ROI). The metrics of concern were schedule, cost and effectivenes s. The research employed an experimental study where subjects worked on individual manua l and automated testing projects. The data collected were cross verified and supplemented with additional data from a feedback survey at the end of the experiment. The result s of this study suggest that automation involves a heavy initial investment in terms of schedule and cost, which needs to be amortized over subsequent test cycles or even subsequent test projects. A positive ROI takes time and any decision to automate should take into consideration the profit margin per cycle and the number of cycles requir ed to break even. In this regard, automation has been found to be effective for testing that i s highly repetitive in nature like, smoke testing, regression testing, load testing and stress testing.

PAGE 10

1 Chapter 1 Introduction This chapter introduces the current software testing scenario and prepares t he foundation for the rest of the study. The layout of this chapter adopts the goal-question metric (GQM) paradigm [1] and the sections are included accordingly. 1.1 Prelude Software defects cost the U.S. economy an estimated $59.5 billion annually or about 0.6 percent of the gross domestic product, according to a study commissioned by the National Institute of Standards and Technology [2]. This cost, unless checked, is bound to grow, since virtually every business today depends on software. The study also states that more than half of this cost is borne by software users and the remain ing by software developers and vendors. The major reason attributed to this cost is an inadequate software testing infrastructure [2]. Recognizing the need, companies worldwide are taking steps to address this issue. This has resulted in a $13 billion software testing market toda y, according to the Gartner Group [3]. Several software testing techniques are in use and one of the most popular techniques is automated testing, primarily due to its ability to execute test cases faster and without human intervention.

PAGE 11

2 1.2 The Problem Software testing may be performed either by manual or automated methods. T he key is in adopting the right technology at the right time under the right circumstanc es. Adopting inappropriate testing techniques may prove expensive in the long run. With software systems growing larger and more complex, testing them becomes i ncreasingly time consuming and difficult. To address this problem, more and more software today uses automated testing to check for defects. However, automated testing, or tes t automation as it is also known, does not always help. Certain tests are better when performed manually. In many cases, manual testing co-exists with tes t automation and the ratio of contribution depends on the testing requirements. Since test automation is at least initially more costly than manual test ing, it should be thoroughly studied before being adopted. The returns on investing in it have to be calculated. Not many Return On Investment (ROI) models for test automation ex ist and those that do, are not vendor independent or backed by empirical evidence. The decision to acquire a test automation tool can be made only if reliable ROI metrics for tes t automation exists. 1.3 The Goals The goal of this study was to determine what expected returns might be obtained when investing in automated testing, and to develop metrics that highlight the various factors to be considered while making the decision to automate testing.

PAGE 12

3 1.4 The Questions In order to assess expected returns for a testing project the following ques tions must be asked: What testing technique should be adopted? Manual or Automated? What are the testing costs associated with this project? How long will the testing project take to complete? How effective is the testing project going to be? 1.5 The Metrics The answers to the above questions may be obtained by identifying, collecting and analyzing the required metrics. Metrics provide an indication of how well a par ticular project is performing. Metrics can also help identify which processes need t o be improved. The key to the success of a project is in the effectiveness of its metri cs [4]. Predictive metrics are of particular value since they may be useful in determining the feasibility of a project before its implementation. This research study ide ntifies predictive return on investment (ROI) metrics for software test automation. During the proposal stages of this study, several metrics were identified: Cost Schedule Effectiveness These metrics are widely accepted in the software industry as factors affecting the ROI on software related projects. The challenge lies in determining how each of t he above metrics actually contributes to software test automation ROI.

PAGE 13

4 1.6 The Experiment This study analyzes the above metrics and the roles they play. It employs a software test automation experiment to help. Twenty four graduate students work ed independently on two testing projects, using manual testing methods for one and automated testing for the other. The experiment was held to a strict timeline and monitored. Sufficient measures were taken to eliminate any bias in the result s. These data and the results from a survey at the end of the experiment allowed comparison of both testing methodologies and an analysis of how various factors affect the adoption of software test automation. The details of this experiment will be discussed in c hapter 3.

PAGE 14

5 Chapter 2 Literature Review This chapter discusses the background material required to understand this study. It discusses software testing in general, software metrics and work si milar to this research that has been done previously. 2.1 Software Testing Software testing is a process used to identify the defects in software a nd to determine whether it conforms to requirements. Although it cannot ensure the absence of defects, testing can at least result in the reduction of defects. Software t esting may either be performed manually or by using automated methods. 2.1.1 Manual Testing Manual testing is a way of testing software where a person creates and exe cutes test cases. The test execution results are compared with the expected output and a ny defects detected are logged. This whole process is performed manually. Several manual testing methods exist today, ranging from simple black box testing to more com plex structural (white box) testing. Some popular methods include Exploratory testing, S tress testing, Load testing, Boundary value testing, Equivalence class testi ng and Risk based testing. Details on these methods can be found in [5].

PAGE 15

6 2.1.2 Automated Testing Automated testing uses software to execute the test cases, compare result s and log defects automatically without human intervention. Automated testing has gained a lot of attention in recent times as a means to reduce testing costs, find more defects and save valuable time. In simpler words it is ‘writing code to test code’. Most test aut omation is performed using specialized testing tools and it requires more specialized s kills than those needed for manual testing. 2.2 Metrics A metric is a standard of measurement. It can be based on a single direct measurement or may be obtained by combining several measurements. A softwar e metric is a measure of its characteristic or property. Some popular software metri cs are Lines of Code (LOC), function points and McCabe’s cyclomatic complexity [5]. A project me tric measures characteristics such as schedule, cost and effectiveness. A g ood metric is defined as one that is valid, reliable, useful and easy to collect. 2.2.1 Metrics for Software Testing Besides using the common software and project metrics like LOC, schedule, et c., certain other metrics are typical to software testing. Defects per LO C, mean time between failures (MTBF) and mean time to failure (MTTF) are among those used to dete rmine the reliability of the code. MTBF is the average time a repairable system will function before failing [4]. MTTF is similar to MTBF except that it is used with reference to systems that

PAGE 16

7 cannot be repaired and the failure is one time only. The system is more reliable i f these numbers are large. Software test automation is basically a software development project by itself because of the planning, implementation and management of test scripts that is re quired. It is used to test the output of another software development project, with an intention to detect defects, improve quality and even reduce cost. Since it takes an investme nt in one software project to improve another, it becomes very important to determine the poss ible ROI before investing in the former. 2.3 Return on Investment (ROI) ROI is defined as the measure of the net income an organization’s management is able to earn with its total assets. Return on investment is calculated by divid ing net profits after taxes by investment. The basic ROI equation [6] [7] is: ROI = Profit Investment ROI is a financial tool that assists management in evaluating how well it can cover its risks. Based on available data, it is used to predict future performance This knowledge may be used to maintain the necessary growth for survival. It is also use d for planning, decision making, evaluating investment opportunities, evaluating current performance and determining responses to the marketplace.

PAGE 17

8 2.4 Related Work Several models have been developed for determining the ROI on software projects and some of popular ones are discussed below. Although they are useful for software projects in general, and definitely have been a source of inspiration f or this study, they cannot be directly applied to test automation. Moreover, ROI models that were developed for test automation tend to focus purely on monetary issues and lack the empirical evidence to back it up. Hence this research. 2.4.1 COCOMO II COCOMO II [8] is an objective cost estimation model developed by Dr. Barry Boehm at the University of Southern California (USC) center for software e ngineering. The model is quite popular and is used for planning and executing software projects. It also supports return on investment decisions. COCOMO II has actually been developed from its precedent COCOMO and its adaptability has been improved so that it can be optimized across various domains and technologies. COCOMO II has two underlying information models. The first is a framework used to describe a software project. This includes process models, work culture, met hods, tools, teams, and the size/complexity of the software product. The second model is an experience base that can be used to estimate the likely resources, for exampl e effort and time, of a project from historical data. This thesis will attempt to study wi th respect to software test automation, some of the factors COCOMO II considers.

PAGE 18

9 2.4.2 Rational Suite Economic Value Model The Rational Suite Economic Value model [9] is used to determine the value obtained from adopting the Rational Suite of tools. It has been developed by Rational a nd is based on COCOMO II. It focuses on metrics like managing complexity, tool experience, use of tools, architecture/risk resolution, team cohesion and process maturity. One of the assumptions made by this model is that that the entire suite of tools is adopt ed and not just the Rational testing tool. While this thesis employs the Rational Robot test tool for the ROI case study, it focuses more broadly on test automation tools in gener al. The implemented version of the Rational Suite Economic Value model can be obtained as an MS Excel sheet by emailing the author (contact details are provided in the s ection ‘About the Author’).

PAGE 19

10 Chapter 3 Experimental Design As stated in chapter 1, the aim of this experiment was to compare manual and automated testing in order to understand the ROI metrics for test automation. The m etrics studied were test automation cost, schedule and effectiveness. This chapter des cribes the experimental design used to conduct this study. 3.1 The Experiment To study the effect of the above mentioned metrics the experiment was designed as follows. A group of subjects was selected for the study and every member of the group performed manual and automated testing on a given software project. Since perfor ming manual testing followed by automated testing might introduce a bias from previously testing the same software and vice versa, two similar projects were conside red (P1, P2), instead of just one. One project was tested using manual testing methodologies (M) and the other used automated testing (A). The group was also separated into two sub groups (G1, G2). There were no separation criteria and the members of each sub group were selected randomly. G1 performed manual testing on P1 and automated testing on P2. Whereas, G2 performed manual testing on P2 and automated testing on P1 (Figure 1).

PAGE 20

11 This way, two different groups worked on the same project (say P1), but one used automated testing and the other used manual testing methods. The results of this experiment have been used to compare the two testing methodologies (manual and automated). A similar comparison was performed with the results from the othe r project (i.e. P2). By cross verifying these two comparisons, unwanted variations were rem oved (Table 1). Figure 1: Experimental Setup The test projects were chosen such that complete test coverage was possible within the project timeline. They were large enough to generate interesting test cases and at the same time, small enough to remain within the scope of this experiment. Two standard, frequently used utilities in the windows environment were selected for functional black box testing namely, the print dialog and the file dialog. G1 G2 P1 P2 M M A A

PAGE 21

12 3.1.1 Print Dialog Testing (P1) Project 1 required the subjects to test the standard windows print dialog. This dialog can be invoked from the menu option: “File > Print…” or by using the shortcut “Ctrl+P” from the application “Notepad” or several other windows based applicat ions. The goal was to come up with a package containing 10 or more test cases that could test the important operations and settings possible using the print utility. 3.1.2 File Dialog Testing (P2) In this project, the standard windows file dialog was tested. It can be invoked from the menu option: “File > Open…” or by using the shortcut “Ctrl+O” from the application “Notepad” or several other windows based applications. The goal here t oo, was to come up with a package containing 10 or more test cases that will test the important operations possible using the File access utility. Table 1: Experimental Setup (G – Group, P – Project M – Manual, A – Automated) Group Project G1 P1 – M P2 – A G2 P1 – A P2 – M The experiment was introduced to the subjects as a class assignment. This assignment was a part of the graduate level software testing course curr iculum at the University of South Florida (USF), in which the students were enrolled. The testi ng assignment document is provided in appendix A.

PAGE 22

13 3.2 The Metrics Choosing the metrics was influenced by the needs of the study as well as the experimental constraints. The study was made possible by the availability o f the Rational Robot testing tool via the IBM Rational SEED program. No monetary investment wa s made to acquire the tool. All cost considerations made for this study are based on licensing costs quoted by IBM Rational [10] [11] and the estimates provided by the Rational Suite Economic Value model [9]. In addition, the effort in terms of person hours was also considered for the cost metric. The project schedule was an important area of focus for this study. The study itself was conducted as a part of the course curriculum and thus had a well defi ned timeline. It was conducted over a period of five weeks, where the first two weeks were allocated for tool training and the remaining three weeks for implementation. The effectiveness metric was subjective in nature. Although the measureme nt of this metric may be debated, the urgent need to study its effect, and the absence of a similar study, encouraged it to be a part of this study. Test automation effec tiveness was measured by studying the number of actual defects and false positives reporte d. Another measure of effectiveness was the response given by the subjects to a survey The survey itself allowed for ratings using five point Likert scales as well as tex tual feedback. The responses were compared to factual data such as the number of person hours put in by the subjects and the actual output generated. This combination provided the productivity details as well as the human factors that influenced it. Finally, it is well known that people are the most important resource of any operation. Measuring their contribution, capability and the factors that affect t hem is vital

PAGE 23

14 to a project’s success. Data such as previous testing experience, productivity and effectiveness user ratings, test coverage, testing scope, testing depth, docum entation and traceability etc. were collected. They have been discussed in chapter 3. 3.3 Data Collection The data was collected over a period of five weeks. A project timeline (Table 2) was followed and the subjects’ progress during this period was monitored. Table 2: Project Timeline Phase Duration Training Two weeks (March 7 – March 20, 2005) Implementation Three weeks (March 21 – April 10, 2005) During the first two weeks the subjects were trained in automated testing techniques. They were first introduced to test automation via a presentation (appendix D) and this included an automated test script demonstration. The sample test script use d for the demonstration is included in appendix D. This was followed by two hands-on lab training sessions that were spaced a week apart. The training documents used for these sessions have been provided in appendix E and appendix F. In addition, the Rational Robot testing tool user guide and the SQABasic scripting language reference guide from IBM Rational [12] were made available for training. The training phase was f ollowed by the three week implementation phase of the experiment, where the test scripts w ere created and executed. The test case template form has been provided in appendix B. At

PAGE 24

15 the end of the experiment, the twenty test cases (10 automated, 10 manual) and defect s (if any) submitted by each of the subjects were collected. The defect reporting template form has been provided in appendix C. These test cases and defects were checked for validit y before being accepted. Finally, a user experience survey was used to collect subjective information. The survey contained a total of 37 questions, ranging from simple Yes/No and 5 point Like rt scale questions, to questions requiring detailed answers. The survey was divided int o four sections. The first three concentrated on one of the three main metrics considere d for this study. The last section focused on other contributing subjective factors. Each of the se responses was cross verified and used to justify the actual experimental re sults. The survey document has been provided in appendix G. 3.4 Other Details This experiment was conducted under strict compliance with the Institutional Review Board (IRB) rules and guidelines at USF (IRB approval # 103422F). The subjects faced no known risks by participating in this study.

PAGE 25

16 Chapter 4 Results & Discussion Demographic data for the experiment has been provided. This is followed by the results. They have been organized based on the three parameters of interest, for whi ch the metrics were collected. Schedule Cost Effectiveness The results under each category are followed by a discussion which includes the inferences made during the study. The test scripts, test case forms, de fect reports and survey data submitted by the subjects for this experiment is available upon request [ 17]. 4.1 Demographic Data The group selected for the study had 24 subjects. 12 were male. The average age of the subjects was 25 years.

PAGE 26

17 0 2 4 6 8 10 12 0 1 year >1 5 years >5 10 years >10 years Programming Experience People (Count) 0 2 4 6 8 10 12 14 16 NoneFormalTested own code Testing ExperiencePeople (Count) Figure 2: Subject’s Programming and Testing Experie nce Nearly 80% of the subjects had less than 5 years of programming experience. A lot of the experience was around 2-3 years. None of them had zero programming experience (Figure 2). Due to the lack of prior software testing experience the 24 subjects could not be categorized by the number of years. Rather the subjects w ere categorized based on whether they had any testing experience involving forma l testing techniques, tested their own code during software development or had no testing experience at all. From figure 2, it is observed that more than 60% of the subjects had no formal software testing experience. About 15% of the test population had a formal experience with software testing. The remaining 25% of them who had experience testing their own code had performed unit and structural tests. Debugging code during softwar e development was not considered as code testing experience. None of them had taken a software testing course before enrolling in the current graduate level s oftware testing course of which this experiment is a part. Except for one individual, no one else had prior

PAGE 27

18 experience with an automation tool. Having limited past training in automation make s it better for the experimental setup since it eliminates any prejudice and e nsures that all feedback from the subject is based on recent experience from the experiment. 4.2 Schedule This section presents the metrics collected for the schedule of the softwa re testing project. Based on the number of builds, a project usually consists of several testing c ycles where each testing cycle performs one complete test of the software. Ba sed on the different tasks performed, a testing cycle may be classified into diff erent phases as follows: Phase 1: Analysis Phase 2: Documentation Phase 3: Tool Training (Applicable to automation only) Phase 4: Implementation (Applicable to automation only) Phase 5: Execution Each phase takes a certain amount of time to be completed. This duration depends on the nature of the tasks performed during the phase. A testing cycle can be represent ed in the form of a timeline, which is actually a graphical representation of all the phases of a testing cycle over time (Figure 3). The length of the timeline is the sum of t he durations it takes to complete each of the fives phases. Timeline length = Analysis time + Documentation time + Tool training time + Implementation time + Execution time

PAGE 28

19 To study the effect of automated versus manual testing on the project schedule; their respective testing cycle timelines can be compared. Data from the experiment is used to construct the automated and manual timelines for the first testing cycl e. It is later projected for the second and third testing cycles in order to observe a trend (if an y). The timelines are cumulative in nature and each subsequent timeline is an extension of t he previous testing cycle’s timeline. After the first testing cycle, based on the nature of testing to be performed, the remaining testing cycles may be further categorized. The automated and m anual testing timelines of the second and third testing cycles are discussed separately under each of the following categories: Regression testing Pure repetitive testing For test automation the phases 1, 2 and 5 are the same as in manual testing. Phases 3 and 4 apply to test automation. This gives us an advance indication that extra effort, cos t and time might be incurred. In a sense, test automation is more of an extension to manual testing. Keeping the current experimental setup in context, the time it would take to ma ke an analysis and come up with 10 test cases is estimated as 1 hour. Similarly the t ime to document the 10 test cases would take another 1 hour. These estimates need not be very accurate for comparison purposes, as phase 1 and 2 are the same for manual and automated testing. According to the data collected from the experiment, the ave rage time to learn the testing tool (phase 3) was 3.85 hours, and the average time to implement the

PAGE 29

20 test cases (phase 4) using the test tool scripting language was 4.01 hours. The avera ge LLOC (Logical Lines Of Code) per user was 140. This means that the time taken t o write a single LLOC is 1.7 minutes, which is reasonable considering it includes the ti me to compile and debug the test script errors. On average, it takes 30 minutes to execute the ten test cases manually. This includes the time to read and interpret the test cases and other human latency. Aut omated test case execution takes around a minute per test case bringing the time fo r 10 test cases to 10 minutes. This is one third of the time it takes to execute the test cases manual ly. Therefore the time saved by automated test case execution is 20 minutes. Now, the time saved is actually more than 20 minutes if we understand what we are concerned with: Actual automated execution time Vs. Time used up on the automated timeline For comparison purposes, we are concerned with the time used up on the automated timeline. The actual automated execution time may be 10 minutes but as long as execution can occur in parallel with other tasks, without human intervention, it does not affect the tester’s timeline. The tester is free to work on the analysis ( phase 1) and documentation (phase 2) phases of the next testing cycle if more software change s are being introduced in the next build. All the tester needs to do is to start the automated execution process and continue with other tasks. So, the time duration that reflects on the timeline is the time it takes to start the automated execution process and it has been estimated as 3 minutes (10% of the manual execution time) for this experiment.

PAGE 30

21 Therefore, the time saved by automated test case execution is now 27 minutes. He nce forth, the automated execution time will refer to the time used up on the timeline. Figure 3 shows the automated and manual testing timelines for a single tes ter. Both of them correspond to the first testing cycle. Comparing manual and automated testing in figure 3, the automated testing timeline is longer than the manual testing timeline by 7.41 hours. This schedule overflow is due to the large training and implementation times for the automated testing cycle (Table 3). 024681012 Manual: 1st Cycle Automated: 1st Cycle Time (Hours) Analysis Documentation Training Implementation Execution Figure 3: 1 st Cycle Time lines – Manual Vs. Automated Table 3: 1 st Cycle Time Periods – Manual Vs. Automated Phase Manual (Hours) Automated (Hours) Analysis 1 1 Documentation 1 1 Tool Training 0 3.85 Implementation 0 4.01 Execution 0.5 0.05 Total 2.5 9.91 4.2.1 Regression Testing Regression testing is the testing of existing code when new code is introduce d. Projecting the above results for regression testing in the second test cycl e (Figure 4,

PAGE 31

22 Table 4), we remove the training time and cut down the implementation time to 10% of the original value. According to industry averages [13], 10% of the original script implementation time is required to maintain and modify the scripts for the next cy cle. For this experiment it is 0.40 hours. It can go up to 15% based on project complexity. The automated and manual timelines for the second testing cycle involving regression t esting are cumulative in nature and are shown in figure 4. 024681012 Manual: 2nd Cycle Automated: 2nd Cycle Time (Hours) Analysis Documentation Training Implementation Execution Figure 4: Regression Testing: 2 nd Cycle Time lines – Manual Vs. Automated Table 4: Regression Testing: 2 nd Cycle Time Periods – Manual Vs. Automated Phase Manual (Hours) Automated (Hours) Analysis 1 1 Documentation 1 1 Tool Training 0 3.85 Implementation 0 4.41 Execution 1 0.1 Total 3 10.36 The numbers in table 4 contributing to the growth of the cumulative timeline in this testing cycle are in italic. We observe a schedule over flow of 7.36 hours for automated testing when compared to that of manual testing. This is lower than the overflow recorded at the end of the first testing cycle which was 7.41 hours. This

PAGE 32

23 indicates that the automated testing timeline is growing at a slower rat e than the manual testing timeline. 024681012 Manual: 3rd Cycle Automated: 3rd Cycle Time (Hours) Analysis Documentation Training Implementation Execution Figure 5: Regression Testing: 3 rd Cycle Time lines – Manual Vs. Automated Table 5: Regression Testing: 3 rd Cycle Time Periods – Manual Vs. Automated Phase Manual (Hours) Automated (Hours) Analysis 1 1 Documentation 1 1 Tool Training 0 3.85 Implementation 0 4.81 Execution 1.5 0.15 Total 3.5 10.81 And by the third testing cycle (Figure 5, Table 5) the automated testing sche dule overflow comes down to 7.31 hours. The decreasing overflow trend as seen in table 6, illustrates the fact that after the first cycle, manual testing is mor e time consuming than automated testing, but the automation timeline is still longer than the manual ti meline due to the large initial investment of time made during the training and implementat ion phases of the first testing cycle.

PAGE 33

24 Table 6: Regression Testing: Automated Testing Cyc le Schedule Overflows Testing Cycle Overflow (Hours) Differential (Hours) Cycle 1 7.41 Cycle 2 7.36 0.05 Cycle 3 7.31 0.05 The overflow gap is steadily decreasing every test cycle and at the curre nt rate of 0.05 hours per testing cycle it would take 147 test cycles before the breakeven point is reached. This is when automation becomes more productive on schedule and ultimately helps ship the product sooner. 4.2.2 Pure Repetitive Testing Sometimes the breakeven point can be attained much earlier than the 147 test cycles required for regression testing. This would be in the case of pure repeti tive testing like, stress, load and performance testing [29]. Here the unmodified test scripts a re run several times on the same software system. These scripts do not require implem entation time as the scripts remain unchanged between test cycles. If you observe the manual and automated timelines for the second testing cycle involving regression testing (Tables 4, 5 and 6), you will notice that the manual timel ine grows at a rate of 0.5 hours per cycle, whereas the automated timeline grows at a rate of 0.45 hours (0.40 for implementation and 0.05 for execution). This time difference of 0.05 hours per testing cycle (Table 6) is what causes the eventual breakeven for regr ession testing. When considering pure repetitive testing, script implementation time is brought down to zero as the software has not changed and therefore no script modifications are

PAGE 34

25 needed. This would eliminate 0.40 of an hour and the automated timeline would only grow at a rate of 0.05 hours per testing cycle. The manual timeline continues to gro w at the rate of 0.5 hours per testing cycle. The time difference between the two t imelines is now 0.45 hours per testing cycle. Let us see the timelines for the second testing cycle involving pure repetiti ve testing (Figure 6, Table 7). Notice that the schedule overflow is now only 6.96 hours instead of the 7.36 hours overflow we observed during the second testing cycle of regression testing. 024681012 Manual: 2nd Cycle Automated: 2nd Cycle Time (Hours) Analysis Documentation Training Implementation Execution Figure 6: Pure Repetitive Testing: 2 nd Cycle Time Lines – Manual Vs. Automated Table 7: Pure Repetitive Testing: 2 nd Cycle Time Periods – Manual Vs. Automated Phase Manual (Hours) Automated (Hours) Analysis 1 1 Documentation 1 1 Tool Training 0 3.85 Implementation 0 4.01 Execution 1.0 0.10 Total 3.0 9.96

PAGE 35

26 Finally, let us compare the manual and automated timelines for the third testing cycle (Figure 7, Table 8). Again, the schedule overflow is only 6.51 hours instead of the 7.31 hours overflow observed during the third testing cycle of regression testing. 024681012 Manual: 3rd Cycle Automated: 3rd Cycle Time (Hours) Analysis Documentation Training Implementation Execution Figure 7: Pure Repetitive Testing: 3 rd Cycle Time Lines – Manual Vs. Automated Table 8: Pure Repetitive Testing: 3 rd Cycle Time Periods – Manual Vs. Automated Phase Manual (Hours) Automated (Hours) Analysis 1 1 Documentation 1 1 Tool Training 0 3.85 Implementation 0 4.01 Execution 1.5 0.15 Total 3.5 10.01 From table 9, we see that the convergence to breakeven point is faster (0.45 per testing cycle) and requires just 15 more test cycles.

PAGE 36

27 Table 9: Pure Repetitive Testing: Automated Testing Cycle Schedule Overflows Testing Cycle Overflow (Hours) Differential (Hours) Cycle 1 7.41 Cycle 2 6.96 0.45 Cycle 3 6.51 0.45 Drawing an estimate based on the Rational Unified Process [14] for a project of this scale, there will be at least one testing cycle each for the implem entation and testing phase of software development. Sometimes there are two testing cycles for each of these phases followed by a user acceptance test during the software development deploy ment phase. So, most organizations on average have between 2 and 5 test cycles before a product release. According to these calculations it becomes clear that automa tion would not benefit this project, be it regression (147 cycles to breakeven) or pure repeti tive testing (15 cycles to breakeven). This outcome was expected as it is quite obvious from the size and scope of this project that we could have easily completed 4 manual test cycles in the time it took to automate the test cases. Figure 8 compares the projected timelines for manual, automated regressio n and automated pure repetitive testing. It is observed that the pure repetitive tes ting timeline reaches the breakeven point much earlier than automated regression testing ti meline. For the sake of clarity, a change of scale has been introduced between the 15 th and 25 th testing cycles on figure 8. This has been done to clearly show the crossover when the breakeven occurs. It is also observed that manual testing has a shorter testing schedule w hen compared to automated regression testing, until the breakeven point is reached at around the 150 th testing cycle. It is therefore vital to choose the testing technique based on the size and scope of the testing project.

PAGE 37

28 0 20 40 60 80 100 120 51015255075100125150175200 Testing CyclesTime (Hours) Manual Auto Regression Auto Pure Repetitive Figure 8: Timeline Comparison – Manual Vs. Regressi on (auto) Vs. Pure Repetitive (auto) The important thing to understand are the factors to be considered while deciding whether to automate or to test manually. These would be, Implementation time + Automated execution time (for automation timeline) Vs. Manual execution time (for manual timeline) They are responsible for the growth of their respective timelines after the first testing cycle. The implementation phase after the first testing cycle mostly perf orms test script maintenance. The implementation time during this phase varies according to Frequency of software changes

PAGE 38

29 Script complexity Level of script reusability Effort required to process test data before script run Effort required to restore system state before script rerun Effort required to restore system state after abnormal termination of the test execution The key is to keep this implementation time low enough so that the automated timeline grows at a rate slower than the manual timeline. In other words, opt for test case automation only if the sum of implementation time and automated execution time is lower than the manual execution time. implementTime auto + execTime auto < execTime man Automation would therefore be a good idea in testing projects where the test scripts are short, simple and reusable i.e. where little effort is spent on imple mentation for test script maintenance. Another point to remember is that the breakeven point needs to be reached within the planned number of test cycles for the project. The difference between the automated timeline and the manual timeline for any testing cycl e gives the current automated testing schedule overflow that needs to be covered in order to break even. Based on the remaining ‘x’ number of testing cycles, 1/x of this overflow should be lesser than the additional growth rate automated testing has over manual test ing in each testing cycle. This difference in growth rate should recover at least 1/x of th e overflow during each of the ‘x’ testing cycles. This difference may be calculated b y subtracting the

PAGE 39

30 sum of the implementation time and automated execution time from the manual execution time. This observation is shown in the form of an equation below. ()()auto auto man man auto execTime ime implementT execTime timeline timeline x + < 1 Where, x = number of testing cycles remaining for the project For example, in our experiment if we had performed pure repetitive testing, at the third testing cycle, we needed 15 more testing cycles to break even. Substituting this and the values from table 8 in the equation above, => (1/15) (6.51) < (0.50 – (0.0 + 0.05)) => 0.434 < 0.45

PAGE 40

31 4.3 Cost There are several kinds of costs incurred when introducing test case automation. They may be classified as either fixed or variable costs [15]. 4.3.1 Fixed Costs The following fixed costs are applicable to test automation: Licensing cost Training cost Infrastructure cost There are several options that may be considered when licensing a tool: Buy vs. Lease – A testing tool may be purchased outright or leased on a monthly or yearly basis if the duration of the project is not certain. The monthly lease m ay be slightly more expensive when compared to the monthly cost if the tool were bought, but it is a good option for pilot projects and customer specific short term testing requirements. For this study, the purchase option was considered, as the goal was t o understand how automation would affect a long term software testing project. Authorized vs. Floating user – By buying an authorized user license, the softwar e is registered to a single user on a single computer and the license may not be transferred. Effectively only one dedicated user may use the software and it cannot be shared. It costs $4120 for one authorized Rational Robot testing tool license [10]. A floating user on the other hand can share the license with other users but only one user may have the license at any one time. This gives more flexibility and would need a licensing server to be setup.

PAGE 41

32 This flexibility comes at a cost. The price for a Rational Robot floating use r license with software maintenance is $7930 [10]. The training cost for this project was calculated as follows. The traini ng phase was spread over two weeks and much of it was covered with two 3 hour hands-on lab sessions. A similar 2 day classroom and hands-on lab training provided by Rational cos ts $1,195 per person [11]. The infrastructure required besides what will be used normally for manual te sting would be a powerful server that can host a licensing server and databases for de fect tracking and test case version control. A server machine costs about $1000. Total fixed costs for this experiment = Licensing cost + Training cost + Infrastructure cost = $7,930 + $1,195 + $1,000 = $10,125 4.3.2 Variable Costs The variable costs applicable to test automation are: Project cost Maintenance cost The project cost is the cost incurred during the various testing cycles of the project. It is dependant upon the project schedule and is calculated as follows:

PAGE 42

33 Project cost = number of person hours cost of tester per hour Where, Number of person hours = number of testers hours worked The maintenance costs could involve hiring consultants for tool expertise, infrastructure maintenance, tool upgrades, tech support etc. or all these can be cove red as a part of the tool vendor licensing contract, at a fixed additional cost. The latter a pplies to this study. Therefore the variable cost for this experiment is simply th e project cost alone. Let us now consider the variable cost of the first testing cycle for this expe riment as shown in figure 9. 68.75 272.525 0 50 100 150 200 250 300 ManualAutomatedVariable Cost ($) Figure 9: First Cycle Variable Cost – Manual Vs. Au tomated Variable cost for the first cycle (automated) = 9.91 27.5 1 = $272.525 Variable cost for the first cycle (manual) = 2.5 27.5 1 = $68.75

PAGE 43

34 Where, Hours worked = 9.91 hours for automated testing, 2.50 hours for manual testing (From table 3). The average wage per hour for a software tester = $27.5 (This data is obtained from the Monster Salary Center [16] and is calculated based on 40*50 work hours per year, excluding holidays). Number of testers = 1 Note: the total cost can be calculated by adding the fixed cost and the variable cost. However, for the sake of clarity, the fixed cost will be discussed separate ly later. From figure 9, it is clear that the variable cost for automated testing is greater than the variable cost for manual testing. This variable cost over flow for automated te sting can be calculated as follows: Automated variable cost – Manual variable cost = $272.525 – $68.75 = $203.775 From the above discussion it is observed that the variable cost overflow is directl y proportional to the schedule overflow. We may calculate the variable cost overflow for the second and third testing cycles by using the schedule overflows calculated in the previous sections.

PAGE 44

35 4.3.3 Regression Testing Since we already have the regression testing schedule overflow values fr om table 6 in section 4.2.1, we can directly calculate the overflow variable costs for the second and third regression testing cycles. This is shown in table 10. Table 10: Regression Testing: Testing Cycle Variabl e Cost Overflows Testing Cycle Overflow (Hours) Cost ($) Cycle 1 7.41 203.775 Cycle 2 7.36 202.4 Cycle 3 7.31 201.025 4.3.4 Pure Repetitive Testing Using table 9 from section 4.2.2, the variable cost overflow for second and third pure repetitive testing cycles can be calculated. This is shown in table 11. Table 11: Pure Repetitive Testing: Testing Cycle Va riable Cost Overflows Testing Cycle Overflow (Hours) Cost ($) Cycle 1 7.41 203.775 Cycle 2 6.96 191.40 Cycle 3 6.51 179.025 4.3.5 Accounting for Fixed Costs As seen in section 4.2, when the automated testing schedule and the manual testing schedule reach a breakeven point after several testing cycles, t he automated variable cost overflow will also be accounted for. This hypothesis is supported by t he steadily decreasing variable cost overflow in table 10 and table 11. However, what is not

PAGE 45

36 accounted for is the fixed cost for automated testing. Considering the best cas e scenario which is pure repetitive testing, the difference between any two consequent va riable cost overflows in table 11 gives the profit margin. This is the cost savings automated t esting has over manual testing, per testing cycle. It is $12.375 for this experiment. Wi th a low profit margin and a high fixed cost ($10,125), it doesn’t make sense to hope for a positive return on investment from the testing cycles of the current project alone. Rathe r, the best option would be to amortize the fixed cost over several testing projects. The averag e profit margin of several testing cycles (spread across several testing projects) can be calculated by summing up their individual profit margins and dividing it by the tot al number of testing cycles minus one. This is shown as an equation below: Average Profit Margin = = + 1 1 1 1 Ti i i (T) cycles testing of Number overflow cost variable Automated Overflow Cost Variable Automated The fixed cost when divided by the average profit margin, gives the minimum number of testing cycles needed to recover the cost. This should be lower than the number of testing cycles an organization plans to complete within a prescribed numbe r of years (n), by which it hopes to obtain a positive ROI. The average time by which an R OI is expected for a software technology is usually three years [9], which is t he time it takes for the technology to become obsolete. This is expressed as an equation below: Project Per Cycles Test Month per Projects n Margin Profit Avg Cost Fixed * < 12

PAGE 46

37 The conclusion drawn is that it is important to consider the fixed cost investment before adopting automated testing. Since the fixed costs are large, it is mor e practical to try and amortize the cost over several testing projects instead of just one. Be sides the average profit margin of the testing cycles belonging to different testing projects should be large enough to amortize the fixed costs within the planned number of years. Thi s should drive the decision to automate. 4.4 Effectiveness The test effectiveness for this experiment is determined by studying th e nature and number of defects found. Several defects were reported but not all were actual defects. Some of them were false positives. Following is a discussion of the inte resting defects detected by manual and automated testing. An attempt is made to rec ognize if a defect pattern exists. 0 2 4 6 8 10 ManualAutomatedDefect (Count) Actual Defects False Positives Figure 10: Actual Defects & False Positives – Manua l Vs. Automated

PAGE 47

38 Table 12: Actual Defects & False Positives – Manual Vs. Automated Classification Manual Automated Actual Defects 5 1 False Positives 9 3 A total of 14 defects (Figure 10, Table 12) were reported via manual testing out of which 5 were valid and 4 defects were reported via test automation, out of which 1 was valid. Here is the defect that was detected during automated testing: The PDF995 print driver that was used to simulate printing while testing the file print dialog did not support multiple page printing and collation. Even if these features were not part of the intended functionality, they should at least be disabled on the fil e print dialog. Here are some of the defects that were detected during manual testing: In the Microsoft Word file open dialog, typing a filename that does not exist and clicking on the ‘open’ button, does not invoke any system response. The user should be informed that the requested file could not be found. In the Microsoft Word file open dialog, the ‘create new folder’ option is enabled. If a new folder is created, it certainly has no files within it that can be ope ned, which makes this option inappropriate for a file open dialog. A recently accessed document is usually listed in the ‘My Recent Document s’ section of a Windows operating system and it can be invoked from this list. Now, delete

PAGE 48

39 this document from its original location and replace it with a different document g iving it the same name as the original. The original document will still be listed in ‘My Recent Documents’ section and trying to invoke it will bring up the new document instead of the original. When large files are attempted to be opened using the file open dialog in the applications ‘Notepad’ or ‘Adobe Acrobat Reader’, the systems hangs if enough me mory is not available to load the file. The applications must handle the situation gracefull y. Memory checks can be enforced and users can be given an option to terminate the ac tion if required. By analyzing the manual defects we get an idea of how devious the manual test cases were [17]. This is why manual testing is highly suited for explorator y testing. The automated test cases in comparison didn’t have much variation. The test cases (scri pts) were modifications of a basic test script [17]. As a result the scripts were performing localized testing. This is probably because scripting takes up valuable test c ase creation time and testers tend to reuse test scripts since it saves a lot of effort. In addition, writing a test script can hinder the creative thinking process. A tester needs to think li ke a programmer and a tester at the same time, which is not easy. On the other hand, it was interesting to see the nature of the false positives detected via test automation. In one case, the tester’s lack of experience with the tool caused him to use the wrong functions in the testing script. The tester was trying to open a text fil e in Microsoft

PAGE 49

40 Word. Failure due to the use of incorrect functions was misinterpreted as faile d functionality. In another case a tester actually embeds the error within the test sc ript unknowingly (faulty test data). When specifying a file name in the file save di alog, it accepts all special characters except ‘*’. For this particular test c ase, every time the test script was run, the test case failed when it encountered the fourth special char acter. It was wrongly assumed that a maximum of only 3 special characters are supported by t he file save dialog. The actual reason was that the ‘*’ character was the fourth cha racter in the test data. The tester failed to recognize this fact because he/she did not han dle the test data during the automated test case execution. The test case itself was c orrect but the defective data led to test failure. There were several false positives detected by manual testing [17]. They ar e not discussed in detail here since they all had a common cause. The tester simply misunderstood the intended functionality of the software. From the above discussion it is clear that manual testing is suited for explorator y testing and in addition to the kind of false positives detected by manual testing, test automation is capable of generating false positives that are testing tool and automation specific. Finally, due to the lack of variation in the automation scripts compared t o the manual test cases, and their very low execution times (discussed in section 4.2) automation will probably work well for high volume repetitive testing like stres s, load and performance testing, rather than exploratory testing. This hypothesis c an be explored during future work.

PAGE 50

41 Note : Neither the DDP (Defect Detection Percentage) or the DRE (Defect Removal Efficiency) metric could be used for this experiment, since only one ma jor testing cycle was performed and an estimate of the final number of defects c ould not be made. 4.4.1 Lessons Learned Be aware of false positives that are testing tool and test automation specifi c. The simple fact that the test case fails and displays an authoritative ‘FAIL ’ log message in red should not psychologically influence the tester to log a defect r eport. Instead, analyze the defect first. Use data pools for testing, wherever possible. It can provide random test data each time the test case is run. Even if it fails once, it has a good chance of succe eding the next time it is run to verify the error. This can reduce the occurrence of data specific errors as observed earlier with an automation false positive. 4.5 Other Metrics In addition to schedule, cost and effectiveness, here are some other observations that were made during the study. These could affect test automation either directly or indirectly. A few of them are subjective in nature. But according to Dreyfus’ model of skill acquisition [18], as a person learns by experience, he/she no longer relies on r ules, guidelines or maxims. Rather he/she has the ability to see situations holisti cally and

PAGE 51

42 intuitively grasps them based on deep understanding. I hope the following discussi on encourages such understanding. 4.5.1 Scope of Testing For this project the subjects found it easy to execute the test cases once the s cripts were ready. Also the application under test did not undergo any changes enabli ng them to use the test scripts repeatedly. As discussed in the previous section, automated t est cases had less variation when compared to the manual ones [17]. They not only were fewer in number, but also exposed fewer defects. Automated testing could be used for creating smoke tests. A smoke test is a nonexhaustive software test to ascertain that the most crucial functions of a sy stem work [20]. The finer details of the system are not tested during a smoke test. A smoke te st can be used to verify the stability of a system before introducing any changes. It can also be used to test the unmodified parts of the software after additional features have been introduced. This will have a better chance at detecting def ect creep as the power of automation is in its ability to execute the test case exactly a s it was intended to. It may be used to stress the system as it is easy to execute the test ca se several times once it has been created. Every automated test case is derived from a ma nual test case. It doesn’t justify the effort in creating an automated test case without manually trying it out first. Unless, it is a performance based test, which usually c annot be done manually.

PAGE 52

43 Automation does not suit exploratory testing. A test case is automated with an intention of repeating it. It loses its novelty when repeated and is not exploratory anymore. It’s only as devious as its manual counterpart. Furthermore, test automation tools today are environment and platform dependant. The Rational tool used for this experiment can be used for automating the testing of Microsoft Windows client/server and Internet applications runnin g under Windows NT 4.0, Windows XP, Windows 2000, Windows 98, and Windows Me. It supports testing of applications developed with IDE’s such as Visual Studio.NET, Java, HTML, Visual Basic, Oracle Forms, Delphi, and PowerBuilder. It does support t esting of custom components but the capabilities are limited. These factors must be conside red while choosing a testing tool. 4.5.2 Depth of Testing This is one area where the project suffered most. Much of the effort was spent in scripting the test cases. Since the project schedule could not be increased, the proje ct scope was reduced. This affected the depth of testing. Also, not all test cases t hat needed automation could be automated due to technical difficulties. In some cases Rationa l Robot did not recognize certain custom controls in Microsoft Word and they had to be tested manually. Verification of graphical output data, such as .PDF files was done manually too. Another challenge was the added overhead of writing scripts to res et the system state every time a test script was executed. This was require d so that the test script could be run again in its original environment. It also ensures that the test script executions are independent of one another, wherever required.

PAGE 53

44 Of the four popular test automation frameworks [21], the test script modularity framework was selected due its simplicity and macro nature. It focuses on the creation of small independent test scripts that represent functions and sections of the applica tion under test (AUT). Keeping it small and function specific enables test case re use. This was important for this project which was small scale and the subjects were relat ively inexperienced with the tool. The test library architecture framework creates a library of procedure s that can be called to test functions and sections of the AUT. It requires creation of an exte nsive library and is justified only if script reuse (not just test case reuse) is ant icipated. It also helps when testers inexperienced with the tool are involved in modifying test script s and a level of abstraction is required. The data driven test framework is quite effective in bringing test variation t o automated test scripts. It involves using variables in the test script which ar e replaced with random values from a database each time the test case is run. Interestin gly, a false positive defect was reported which would not have been if this framework had been adopted. Details discussed in section 4.4. Finally, the keyword driven architecture involves the creation of a keyword tabl e with the columns like ‘Window’, ‘Control’, ‘Action’ and ‘Arguments’. The script picks up details of the action to be performed on a control in a particular window from this table. This would require an extensive inbuilt script library to exist. Besides the testing framework, the recognition capability of the tool also pla ys an important role. Screen controls are recognized in two ways [21]: Recognition by screen coordinates

PAGE 54

45 Object recognition Object recognition is better of the two and gives greater control over the de pth of testing. Recognition is by a unique property of the control and is irrespective of it s position on the screen. Recognition by screen coordinates does not give access to the properties and state of the control. 4.5.3 Test Tool Adoption Adopting a test tool is a major challenge when shifting from manual to automat ed testing. Test automation involves test case scripting and this requires a cert ain level of programming experience, without which the testers cannot understand IF, FOR and other programming constructs used in the scripts. Manual testers by default are not expected to be experienced in programming. According to the experiment’s survey (Figure 1 1), this is how the subjects in this experiment thought prior programming experience affected t heir performance on a scale of 1 (negligible) to 5 (very high). Those with no prior programming experience, answered ‘N/A’. 0 1 2 3 4 5 6 7 NegligibleLowMediumHigh Very HighPeople (Count) Figure 11: Effect of Programming Rating

PAGE 55

46 Since most of the subjects had 2-3 years of programming experience (Figure 2), they were comfortable with the concept of test case scripting. In addition, the GU I recording option of the testing tool made learning easier. The subjects could auto-generate scripts and modify them according to requirements. However, the s cript creation process was time consuming. Subject reports of script writing experience are shown in figure 12. 0 2 4 6 8 10 Very easyEasyModerateDifficultVery DifficultPeople (Count) Figure 12: Scripting Experience If the available testers lack programming experience, a dedicated test automation programmer can be hired. Although it helps, this person need not be a tester. His/her responsibilities would be to automate the test cases rather than thinking them up. This person may be shared across several projects.

PAGE 56

47 0 2 4 6 8 10 12 14 PoorFairGoodVery Good ExcellentPeople (Count) Figure 13: Training Effectiveness Rating Besides prior programming experience, adequate tool training also plays a vital role in a successful test tool adoption. For this experiment, training was provided over a period of two weeks with several hands on lab sessions (details in chapter 3). Subject feedback on the effectiveness of training is shown in figure 13. 0 2 4 6 8 10 12 Very easyEasyModerateDifficultVery DifficultPeople (Count) Figure 14: Test Tool Usability Rating Finally, the usability factor of the tool also contributes to test tool adoption. If a tool is user friendly, it has a low learning curve. The subjects found the Rational Robot testing tool used in this experiment user friendly and intuitive (Figure 14). They w ere able to successfully adopt and use the tool within the planned two week training period.

PAGE 57

48 4.5.4 User Ratings for Effectiveness & Productivity 0 2 4 6 8 10 12 PoorFairGoodVery Good ExcellentPeople (Count) Figure 15: Effectiveness & Productivity Rating The user rating on the effectiveness and productivity of test automation is shown in Figure 15. Subjects referred to test automation as an important skill in the marke t. One subject said he did not find much use for automation. Actually, the subjects didn’t see the benefit for the small project they were working on and believed it would do better for regression testing (point highly stressed). They also felt that it improve d test coverage and eliminated human error. Finally although, automation cannot replace human creativity, it is faster and the amount of productivity depends on what is being automa ted. It was interesting to observe a good rating on test automation effectiveness a nd productivity even though the number of defects found using automation was far less than with manual testing. Much of the optimism was based on projecting its capability on a project of larger proportions that would involve regression testing.

PAGE 58

49 Furthermore, it reflects the high confidence in automation among users and customers. Surprisingly this is a primary reason why many organizations adopt automation. It gives them a unique selling point and buy-in from the customer. 4.5.5 Documentation and Traceability The effort subjects put in for documentation is as follows. 0 2 4 6 8 10 12 14 Very easyEasyModerateDifficultVery DifficultPeople (Count) Figure 16: Documentation Effort Clearly, effort has been put in to document the testing process (Figure 16) and rightly so. Test documentation is vital for measuring test coverage, tracking ch anges and managing defects. Test automation aids this process by maintaining logs an d generating reports. This information may be used to automate documentation and reduce manual effort. Automated test case execution may be tracked and test failures may a uto generate defect reports. This can improve traceability. In this regard, not just test execution but the whole testing process may be automated. Integrated tools [23][24] exist for documentation, defect tracking, requirements management, test coverage, configuration management etc.

PAGE 59

50 4.5.6 Automation – Individual Vs. Group Effort This experiment involved 24 individual test automation and 24 individual manual testing projects. More than 65 % of the subjects recommended Individual work. The subjects appreciated the hands on experience and felt that the dedicated individual e ffort ensured better learning. They also noted that groups would enable sharing of ideas. The test cases and testing approach may be discussed in groups whereas the script implementation may be performed individually. This may be considered analogous to a software development project, where coding is an individual effort. Even the pair programming paradigm [33] recommends one person think up the approach/solution and the other implements it. Thinking and execution don’t seem to be possible in parallel and hence the pair programming concept. 4.5.7 Test Coverage Based on a survey on the thoroughness of testing achieved, 44% of the subjects observed that they could test more thoroughly using automation. Automation ensured that they didn’t miss any steps during test case execution. Some quoted that the initial tool learning curve, took up time and prevented them from testing thoroughly. Two inferences can be derived from the above observation. Firstly, automation ensures the test case is executed exactly the same way each time. Somet imes testing gets mundane and humans tend to skip test case steps. This doesn’t happen with automation. There is little scope for human error. Automated tools can also be used to track exe cution of test paths. Some tools can generate all possible program paths based on which the t est

PAGE 60

51 cases may be created. Executing the test cases and tracking them using t he automated tool helps monitor test coverage. The results, inferences and lessons learned in this chapter set the stage for f urther discussion. This information may be used to estimate the behavior of an automation project and how automation in general compares to manual testing. Of course all thi s may be done keeping in mind the limitations as well as contributions of this experiment towards this research study. 4.6 Tripod SDLC Model Manual testing and automated testing are quite different from one another and comparing them is like comparing apples with oranges. Although both are based on a common testing paradigm, one is based on a creative process, whereas the other is strongly influenced by the software development life cycle. This made it diff icult to come up with a statistical comparison. Choosing one and eliminating the other was not an option. Therefore, for an effective testing setup, it is important to analyze the r oles they play and adopt them for the right testing requirements accordingly. Based on this research study and the experience of others in automation [19], it is clear that the effort required to adopt test automation has been widely underest imated. This is probably why there are so many failed attempts at test automation even t hough it is quite a useful technique. In this light, I would like to propose a modified V-model software development life cycle, called the tripod model (Figure 17).

PAGE 61

52 Figure 17: Tripod SDLC Model The tripod SDLC model is basically a modified Vmodel that has been customized for software test automation. Here test automation has been treated a s a dedicated development path with its own corresponding requirements, analysis and design phases. This is required as the automation effort today can be compared to software development with its own scripts that are implemented, tested, managed, ve rsion controlled and maintained with frequent updates and upgrades. The various phases of test automation are discussed below: 4.6.1 Automation Requirements In this phase, the customer requirements for automation, like smoke tests, user acceptance tests (UAT), etc. are gathered and documented. The training, lic ensing, and other infrastructure details of automation are taken care of in this phase. The customer business rules are studied to understand how automation can be applied effectively to t est the software. This phase lays the foundation for automation in the following phases

PAGE 62

53 4.6.2 Automation Feasibility Here, the test cases created as a part of the ‘analysis’ phase of the softwa re SDLC are inspected to see if they can be automated, given the current system envir onment, testing methodologies and available testing tool. Schedule constraints and test c ase priority are also considered. The feasible test cases are short listed for automation. 4.6.3 Framework Design Based on the test cases short listed in the previous phase and the software design created in the ‘design’ phase of the software SDLC, an appropriate test framew ork is chosen for test automation. Depending on the need, it could either be the test script modularity framework, the test library architecture framework, the data dr iven test framework or the keyword driven architecture framework. 4.7 Contributions Extended schedules, budget overruns and poor test results are often encountered when dealing with test automation. This continues to happen even when current ROI calculators (Chapter 2) project very high ROI percentages for the test pr oject. This thesis explores the real reasons behind this unfavorable pattern. Understanding the streng ths, weaknesses and amount of effort required for automation can go a long way in preventi ng these failures. The main contribution of this research is the detailed explora tion of the three main metrics that would be considered for an ROI study of software test automation. These are schedule, cost and effectiveness. Recommendations based on empirical evidence have been made in the form of equations for these metrics. Us ing

PAGE 63

54 these equations as guidelines while adopting automation should give a good return on investment. Other factors that could affect a test automation project have been discussed, eve n though some of them were subjective in nature. It was more of an attempt to understand them and attain a holistic view. Finally, a modified V-model SDLC called the Tripod-model has been proposed. It should hopefully cater to current and future software development projects where test automation is becoming an integral part of the development and testing cycle. 4.8 Limitations Just as it has interesting contributions, this study also has its limitations. Unfortunately none of them were avoidable under the circumstances and scope of this study. They are discussed below with the hope that others doing similar research can avoid them or plan accordingly. They also serve as a check list for things to consider i n my future work. 4.8.1 Project Scope The scope of the project was limited. It had to be reduced when the subjects found it too difficult, especially the subjects who were being introduced to automation for the first time. An alternative solution of extending the schedule was not possible as the study was a part of a course curriculum.

PAGE 64

55 4.8.2 Error Seeding Again due to schedule considerations, we had to test readily available software instead of developing it ourselves first. This eliminated any chance of error seeding to that would have further helped in determining the effectiveness of automated te sting. By choosing relatively simple software for testing, based on considerations like use r experience, tool environment support and time, the number of defects turned out be very low. Without a good sample, analysis becomes difficult. 4.8.3 Automation Tool The research study was limited to the Rational Robot testing tool. Thanks to the IBM Rational SEED program [15], this tool was made available free of c ost in support of educational research. Although Robot is a well recognized representative of all testing tools on the market today, without actual comparison with a similar test ing tool, some of the biases that are tool specific may not be removed. Testing was also res tricted to environments and platforms supported by the tool.

PAGE 65

56 Chapter 5 Conclusion In conclusion, the goal of studying the major factors affecting test automa tion, namely schedule, cost, effectiveness and developing metrics for ROI calcul ations has been met. The detailed discussions on the above factors have been covered in chapter 4. A summary of the study results is provided in the form of recommendations (Table 1 3). They have been designed to maximize ROI. One may choose either automation or manual testing for a project based on these recommendations. For details please re fer to chapter 4. Table 13: ROI Recommendations Automated Testing Manual Testing Schedule Implementation time + Automated execution time < Manual execution time. {(1/x)* (Automated timeline – Manual timeline)} < {Manual execution time – (Implementation time + Automated execution time)}. Rapid User Acceptance Testing (UAT). Already late projects.

PAGE 66

57 Cost Multiple testing projects to amortize fixed costs. Average profit margin = T-1 (Automated variable cost overflow (i) – Automated variable cost overflow (i+1) ) / Number of testing cycles (T) -1 (Fixed costs / Average profit margin) < n* 12*Number of projects per month*Number of testing cycles per project. Low Training budget. Low Infrastructure budget. Effectiveness For Smoke testing, Regression testing, Stress testing and Load testing. For multi versioned software products. For Repetitive testing. Frequently upgraded software. High testing redundancy. For Exploratory, Functional testing. For software services. For testing with fewer test cycles. Frequently modified software. 5.1 Future Research In this study, the effect of the second and consecutive test cycles have been projected based on the behavior of the first. Industrial averages have been used in the calculations wherever data was unavailable. All this is attributed to the reas on that the experiment was completed under a tight schedule with no scope for more than one test cycle. The test population was relatively small too. For the sake of reproducibil ity and verification it would be preferable to perform the study again without the above limitations and the ones mentioned in section 4.8. Two factors that strongly influenced the cost and schedule during this study were the initial investments made for training and test script implementation. Any positive returns test automation brought was spent in recovering these initial investme nts. These

PAGE 67

58 investments could have been specific to the tool we used. In the future, it would be a good idea to perform a case study of available test automation tools. Going further the future of automation tools can be explored based on what the current tools lack. The basic need is to cut down on training and implementation investments and this can be achieved by making tools easier to use and auto-generate test scripts. Certain tools do auto generate test scripts today, but they tend to recognize controls on the screen using screen coordinates and the tools that recognize controls as objects are not very effici ent. The underlying problem is that whether you auto-generate your test scripts or w rite them manually, when the software code changes, the test scripts have to change. So, b y deriving a metric that measures software code change between test cyc les, ROI calculations can be made more accurate. Alternatively this problem may be handl ed by exploring testing frameworks that totally eliminate scripting. A step in t hat direction is the keyword driven testing framework [21]. Finally, this study is based on functional testing. Future work can try and relate these results with testing based on security, stress, load and performance.

PAGE 68

59 References [1] V.R. Basili, “Software Modeling and Measurement: The Goal Question Metric Paradigm,” Computer Science Technical Report Series, CS-TR-2956 (UMIACS TR-92-96), University of Maryland, College Park, MD, September 1992. [2] RTI, “The Economic Impacts of Inadequate Infrastructure for Software Tes ting”, NIST, Planning Report 02-03, May 2002. [3] The Garner Group, URL: http://www.gartner.com [4] N. Fenton, R. Whitty, Y. Iizuka, “Software Quality Assurance and Measurement: A Worldwide Perspective”, International Thomson Computer Press, 1995. [5] P. C. Jorgensen, “Software Testing: A Craftsman’s Approach”, Second Editi on, CRC Press, 2002. [6] Six Sigma SPC, “Quality Control Dictionary”, URL: http://www.sixsigmaspc.com/dictionary/glossary.html [7] R. Rachlin, “Return On Investment Manual: Tools and Applications for Managing Financial Results”, Sharpe Professional, 1997. [8] B. W. Boehm et al, “Software Cost Estimation with COCOMO II”, Prentice Ha ll PTR, 1 st edition, 2000. [9] J. Barnes, “Creating an ROI assessment for implementing IBM Rational s olutions”, Rational DeveloperWorks, 15 Oct. 2004, URL: http://www128.ibm.com/developerworks/rational/library/content/RationalEdge/oct04/barnes / [10] IBM Rational, “Rational Robot Licensing”, 2005, URL: https://www112.ibm.com/software/howtobuy/buyingtools/paexpress/Express?P0=E1&part_num ber=D53NDLL,D52ZFLL&catalogLocale=en_US&Locale=en_US&country=US A &S_TACT=none&S_CMP=none

PAGE 69

60 [11] IBM Rational, “Rational Robot Training”, 2005, URL: http://www304.ibm.com/jct03001c/services/learning/ites.wss/us/en?pageType=course _descript ion&courseCode=RT511#2 [12] IBM Rational, “Rational Robot User Guide”, Version 2003.06.00, 2003, URL: http://www.rational.com [13] Mentora Group, “ROI of an Oracle Apps Repeatable Automated Test Suite”, URL : www.mentora.com/library [14] IBM Rational, “The Rational Unified Process (RUP)”, URL: http://www.rational.com/rup/ [15] D. Hoffman, “Cost Benefits Analysis of Test Automation”, STAR West 1999, Oct 1999. [16] Monster Salary Center, “Median Salary for a Software Quality Analy st”, 2005, URL: http://salary.monster.com/ [17] N. Jayachandran, “Compiled Test Scripts, Test Cases and Defect Reports Fr om The Experiment Conducted For This Thesis”, 2005, Email: naveenjc@yahoo.com (Available upon request). [18] H. L. Dreyfus, “A Phenomenology of Skill Acquisition as the basis for a Merleau Pontian Non-representationalist”, University of California Berkeley. [19] M. Fewster, D. Graham, “Software Test Automation: Effective Use of Test Execution Tools”, Addison-Wesely, 1999. [20] search390.com, “Definitions – Smoke Test”, 2003, URL: http://search390.techtarget.com/sDefinition/0,,sid10_gci930076,00.html [21] M. Kelly, “Choosing a test automation framework”, IBM DeveloperWorks, 18 Jul 2003, URL: http://www-106.ibm.com/developerworks/rational/library/591.html [22] J. LaRogue, “IBM Rational SEED Program”, The Rational Edge, 2004, URL: http://www-128.ibm.com/developerworks/rational/library/2273.html [23] IBM Rational, “Rational Suite”, 2005, URL: http://www306.ibm.com/software/awdtools/suite/ [24] Mercury, “Mercury TestDirector”, 2005, URL: http://www.mercury.com/us/products/quality-center/testdirector/works .html

PAGE 70

61 [25] J. W. Spechler, “Managing Quality in America’s Most Admired Companies”, Berrett-Koehler Publishers, 1993. [26] Task Committee on Software Evaluation, “Guide for Evaluating Software Engineering”, ASCE, 1989. [27] B. C. Meyers, P. Oberndorf, “Managing Software Acquisition: Open Systems and COTS Products”, Addison-Wesley, 2001. [28] R. S. Pressman, “Software Engineering: A Practitioner’s Approach”, Sec ond Edition, Mc-Graw Hill, 1987. [29] C. Kaner, J. Falk, H. Q. Nguyen, “Testing Computer Software”, Second Edition, Thomson Computer Press, 2001. [30] S. H. Kan, “Metrics and Models in Software Quality Engineering”, Second Edition, Addison-Wesely, 2004. [31] G. G. Schulmeyer, J. I. McManus, “Handbook of software Quality Assurance”, Macmillan, 1987. [32] V. Sikka, “Maximizing ROI on Software Development”, Auerbach Publications, 2005. [33] L. Williams, R.R.Kessler, W. Cunningham, R. Jeffries, “Strengthening the Cas e for Pair-Programming”, IEEE Software, 2000. [34] J. Varghese, “Test Automation – An ROI based Approach”, 3 rd Annual International Software Testing Conference, Bangalore, India, 2001. [35] J. Bach, “Test Automation Snake Oil”, 14 th International Conference and Exposition on Testing Computer Software, 1997.

PAGE 71

62 Appendices

PAGE 72

63 Appendix A: Experiment ISM 6930 – SOFTWARE TESTING SEMINAR SPRING SEMESTER 2005 TESTING ASSIGNMENT 5 – Due: April 11, 2005 Automated Testing vs. Manual Testing 1. Assignment Now that you are familiar with manual and automated testing methodologies, this testing assignment will train you to judge when to choose automated testing over manual methods and vice versa. For this assignment, the class will be split into two equal groups (G1, G2), where the group members will be selected according to the alphabetic al order of their last names. The groups are: Group 1 (G1): Bassi, Cao, Chandra, Connor, Douglas, Gibson, Gowda, Guduru, Hadadare, Jordan, King, Magnusson, Marudur Srinivasan, McCart Group 2 (G2): Morrill, Nemakal, Nidamarthi, Ostapenko, Pasupathy, Rakasi, Senarathne, Shahasane, Srivastava, Tatiparthi, Tello, Warrilow, Yassin, Yussouff Each group will work on two sub-assignments over a period of two weeks as follows (Fig. 1): Group Sub-Assignments G1 M-1, A-2 (M – Manual, AAutomated) G2 M-2, A-1 Therefore, both groups will work on one manual testing sub-assignment and one automated testing sub-assignment each. The above arrangement is used to avoid an y testing bias.

PAGE 73

64 Appendix A: (Continued) Fig 1: Experimental setup 1.1 Testing Sub-assignment 1 (Print Dialog) You will be testing the standard windows print dialog. This can be invoked from the menu option: “ File > Print… ” or by using the shortcut “ Ctrl+P ” from the application “ Notepad ” or several other windows based applications. The goal is to come up with a package containing 10 or more test cases that will test the important operations a nd settings possible using the print utility. Testing Sub-assignment 2 (File Dialog) In sub-assignment 2, you will be testing the standard windows file dialog. This ca n be invoked from the menu option: “ File > Open… ” or by using the shortcut “ Ctrl+O ” from the application “ Notepad ” or several other windows based applications. The goal here too, is to come up with a package containing 10 or more test cases that will test the important operations possible using the File access utility. Note: The advantage of the automation exercise is that you may import and reuse these s cripts when testing the print or file access functionality of any application in the future. G1 G2 Sub Assignment 1 Sub Assignment 2 M M A A

PAGE 74

65 Appendix A: (Continued) 2. Automation guidelines 2.1 Read through the Rational Robot: User’s Guide available in the downloads section of blackboard. 2.2 Write down the tests you would like to perform on the module as simple one line sentences. You may use any functional testing paradigm to develop your test c ases. Remember to be devious in your selection of test cases. (You may later use them to create your test scripts and even add more detail. It is advisable to keep your sc ripts well documented and maintainable.) 2.3 Use the object recording feature to give you an idea of the functions you can use t o perform an operation. Then select the desired function and press ‘F1’ to learn how to use the selected function in your own script. 3. Deliverables 3.1 You are expected to deliver at least 10 test cases each, for sub-assignments 1 a nd 2 (A total of 20 or more). Report each test case using Functional Test Case Report ing Forms attached to this assignment. 3.2 You also need to submit the test scripts you create for the above test cases, usi ng the tool, for A-1 or A-2, according to the group you belong to (G2 or G1, respectively). Include comments in your scripts to make it easier to understand and separate one t est case from the other. 3.3 Complete Defect Reporting Forms if running a test case results in a failur e. Report the defect using the report form attached. 3.4 Finally, you need to submit a completed copy of a survey, which will be provided during the lab session on March 28. 4. Assignment timeline The following timeline is recommended so that you may complete your assignm ent comfortably and efficiently. 4.1. Reading phase: (March 7, 2005 to March 20, 2005)

PAGE 75

66 Appendix A: (Continued) Two weeks to go through the Robot documentation, get used to the Robot editor and create simple scripts. Last date for queries and clarifications on the above i s March 20, 2005. 4.2. Working phase: (March 21, 2005 to April 11, 2005) Three weeks to create test scripts and complete the assignment. Last date for queries and clarifications on the above is April 1, 2005. 4.3. The assignment is due by 6pm on Monday, April 11, 2005. Please submit your deliverables as a single compressed folder, via blackboard. I will be available to answer your queries regarding this assignment at naveenjc@yahoo.com 5. Lab: Getting started 5.1. Rational license server: ibmlab5 5.2. Common space on the network has been provided so that you create, modify and store your project online. This space has been provided as a hidden network drive. To access this space: You need to map a network drive. Right click on ‘My Computer’ and select ‘Map Network Drive…’ In the window (dialog) that pops up, enter the following information. Drive: H Folder: \\bsnlab\hevner$ Note: 1. Remember to uncheck the checkbox ‘Reconnect at logon’. If you fail to do this, other students who are not a part of this course can access this drive when they logon using the same computer. 2. Create a new directory within the H: and give it your name. Your working and stora ge space must be within this directory only. This is to ensure that users do not accidentall y overwrite other user’s files.

PAGE 76

67 Appendix A: (Continued) 3. When you create a Robot project (will be discussed in class), password protect it, so that other students who can access the common network drive, cannot copy your test scripts. 4. You are personally responsible for your files. Please take a backup of your test script files on a floppy disk, after each Robot session. If you have a flash drive, you may ta ke a backup of the entire project, each time (will not exceed 15 MB). 5. You are given full rights (read, write and execute) on this network drive to enable hassle free working on the assignment. All activities on this network drive are logged.

PAGE 77

68 Appendix B: Functional Test Case Reporting Form Test Case ID Component Purpose of Test Case Functional Test Type Pre-Conditions Inputs Expected Outputs Post-Conditions Execution History Date Result Tester Test Case ID Component Purpose of Test Case Functional Test Type Pre-Conditions Inputs Expected Outputs Post-Conditions Execution History Date Result Tester Test Case ID Component Purpose of Test Case Functional Test Type Pre-Conditions Inputs Expected Outputs Post-Conditions Execution History Date Result Tester

PAGE 78

69 Appendix C: Defect Reporting Form DEFECT ID: TITLE: COMPONENT: REPORTED BY: SUBCOMPONENT: REPORTED ON: VERSION: PLATFORM: RESOLUTION PRIORITY: OPERATING SYSTEM: RESOLUTION SEVERITY: RELATED TEST CASE IDS: CONSIST ENCY: DEFECT SUMMARY: STEPS TO REPRODUCE: COMMENTS: ATTACHMENTS: 1. 2. For evaluators use only DEFECT STATUS: _________APPROVED _________ REJEC TED REASON: ________________________________________________

PAGE 79

70 Appendix D: Tool Training Presentation Slides nnrnn Software Testing SeminarClass 9 –Automated Testing Tools and Orientation to IBM Rational Testing Tools r !"#!r$r%&'#(r# )r$%r*r# '&*+ nnrnn Class 9 Outline r#rrr&#' r''rr r,r -&rr

PAGE 80

71 Appendix D: (Continued) nnrnn Automated Testing Key Points )##rr$r&#' )r&#'$r)r#%&' 'r&r% /0,r /r##1 r00,#r## /r#,$1 +rr2rr&#'&$r$r3 nnrnn 4 More Key Points 'rrr##r# -rr50#r6rr$0#r&, *rrrr$0#r&$ r&#r&$r0r#/2r -r#'&r2$rr&#' '&07r#8

PAGE 81

72 Appendix D: (Continued) nnrnn Advantages of Automated Testing *r +#r 9$$r$&#r r &r#r#$$r nnrnn : Are You Ready for Test Automation? 'rr;
PAGE 82

73 Appendix D: (Continued) nnrnn Automated Testing Risks 'r&r%r$0-r& 'r&r+r '&& =rr>(r#%2 !r? %@ r %+ %%& A=r+ nnrnn B Capture/PlayBack rrr$r$rr '$r&&r#rn 0&$ r 'r&#r@r##r$C,2r&,& $r&, r## $&2r#&, r##

PAGE 83

74 Appendix D: (Continued) nnrnn D Class 9 Discussion Questions r00r#rr3&, r&#rrrrr31Arrrr0 /0$##r&#rr1 r0,r#r$0r3,7r$0@#&,r$$$8,$rr&r3$#1 nnrnn n IBM Rational Testing Tools Orientation Outline r&#' rr,r -&rr0 -&r '& r&&### rr +,>A#

PAGE 84

75 Appendix D: (Continued) nnrnn Hierarchy %r$0 9 %r$0 -r& %r$0 ' r&# nnrnn Automated Testing E *rr7 rr,r /8 +r#%7/%'+r#8 )7")r($r( r8 Errr r#F,2 % r E%, +% --% G0r#-% Err, %rr# ?,3rr

PAGE 85

76 Appendix D: (Continued) nnrnn Rational Robot ,rrHr r&r rr,r r' %Ir# ?,3rr I$rr -(rr r#r$ )$rr r ?,3& nnrnn 4 Demo overviewJ (-r' 2r$#;#r& ?;#r&7$#r&r#r8 )$&rr2#r 4 2r$#5+#6rr7?,3rr8 I$#rr : r$#r (#r& B r(#r

PAGE 86

77 Appendix D: (Continued) nnrnn Testing Assignment 5 nnrrn A A /7H r %, & %, & nnrnn : Recommended Reading r,rK'r&rL%r$0'#M9%&,C?r,nnn7 E2Er# 8 GNK=r#$r'r&rA&1L%r$0'#M9!r&,C-&,nn7 E2Er# 8 2*0#-rrA&K%r$0'r&rH9$$r$;rrrL rr,r)A#7 E2Er# 8 %ME$7 E2Er# 8

PAGE 87

78 Appendix D: (Continued) The Rational Robot test script for the ‘Demo Overview’ presentation slide in Appendix D is included below. The slide describes the actions of the test script. This test s cript was used to train the subjects and gives an idea about the automated test scripts wri tten by the subjects for the experiment. Sub Main Dim Result as Integer Dim returnValue As Integer Dim attrValue as Integer Dim msgBoxText as String Dim value as String On Error Goto ExceptionHandler attrValue = Getattr(“C:\Documents and Settings\Nav\Desktop\RobotDemo.txt”) SQAConsoleWrite “File Attribute is: “ & attrValue If attrValue = 0 then SQAShellExecute “C:\Documents and Settings\Nav\Desktop\RobotDemo.txt”,””,”” Window SetContext, “Caption={*Notepad}”, “” MenuSelect “File->Print...” Window SetContext, “Caption={*Print}”, “” PushButton Click, “Text=Preferences” Window SetContext, “Caption={*Preferences}”, “” RadioButton Click, “ObjectIndex=2” Result = SQAGetPropertyAsString(“Type=RadioButton;ObjectIndex=2”, “State”, value) SQAConsoleWrite “State is: “ & value Window CloseWin, “Caption={*Preferences}”, “” Window CloseWin, “Caption={*Print}”, “” Window CloseWin, “Caption={*Notepad}”, “” Msgbox “Test Run Complete” Exit Sub Else Exit Sub End if ExceptionHandler: msgBoxText=”Error code: “ & Err & “ Error line: “ & Erl MsgBox msgBoxText

PAGE 88

79 Appendix E: Testing Tool Lab Training 1 ISM 6930 – SOFTWARE TESTING SEMINAR SPRING SEMESTER 2005 TESTING ASSIGNMENT 5 Rational Robot Exercise 1 – March 21, 2005 This exercise has been designed to get you started with the basic Robot commands The functionality of most of these commands is evident from the command name itself. Your task is to create a test script and explore the commands listed below. Getting Started 1. Make sure the H: drive is mapped to \\bsnlab\hevner$ 2. Open Rational Robot. (Start > All Programs > Rational Software > Rational Robot 3. If you can’t find your project in the drop down list, then you need to register it using Rational Administrator (Open Rational Administrator and use the option ‘Register Project’. If you are unsure, ask me how). 4. Once you are within Rational Robot, with your project open, you need to create a new test script (File > New > Script…). 5. In the script editor that opens up, you can type in the commands given below. Place your cursor on the command you are interested and press F1. A comprehensive help dialog opens up. The help dialog explains the command syntax and usage. It usually also has example scripts that you can refer to. Explore and enjoy! Command List 1. Dim (Also read about the various data types) 2. Window SetContext 3. PushButton Click 4. InputKeys 5. ComboEditBox 6. ComboBox 7. ComboEditBox 8. ListBox 9. CheckBox 10. MsgBox 11. DelayFor 12. SQAGetProperty

PAGE 89

80 Appendix E: (Continued) Note 1. To explore controls that you do not already know the details of, use the Inspector tool (Tools > Inspector…). After invoking the tool, you might have to wait for a few seconds for it to evaluate all screen objects and initialize. After it c omes up, you can click on the hand icon at the lower right hand corner of the tool, drag and drop it on the control you wish to use. The tool will provide you with all the details you need about the control. 2. You may also use the Record and Playback tool (discussed in the previous class – March 7) to get an idea of how the commands are used. Tip 1. When looking up help on a command, click on the link ‘See Also’ on the top left hand corner to explore related commands.

PAGE 90

81 Appendix F: Testing Tool Lab Training 2 ISM 6930 – SOFTWARE TESTING SEMINAR SPRING SEMESTER 2005 TESTING ASSIGNMENT 5 Rational Robot Exercise 2 – March 28, 2005 The goal of exercise 2 is to use some of the commands you learned in exercise 1 to convert your TA5 test cases to automated scripts. Below you are provided with t wo sample scripts one for each group (G1, G2), to get you started. So, today you will, Create, execute and understand the two sample scripts. Think of similar test cases for TA5 and try to automate them based on what you ha ve learned. 1. G1: File Dialog Testing (A2) 1.1. Sample Test Case Open a text file by specifying a path in the file dialog. 1.2. Sample Test Script Sub Main SQAShellExecute “C:\windows\notepad.exe”,””,”” InputKeys “^{o}” Window SetContext, “Caption=Open”, “” ComboEditBox Click, “ObjectIndex=3”, “” InputKeys “{bksp 5}” InputKeys “C:\” PushButton Click, “Text=Open” End Sub 1.3. Note 1.3.1. Make sure you create a text file and use the correct path in the script. 1.3.2. Try and modify the above script so that it automatically closes the text fil e to avoid having to manually do it each time. 2. G2: Print Dialog Testing (A1) 2.1. Sample Test Case Print a text file using the print dialog.

PAGE 91

82 Appendix F: (Continued) 2.2. Sample Test Script Sub Main SQAShellExecute “C:\”,””,”” Window SetContext, “Caption={*Notepad}”, “” MenuSelect “File->Print...” Window SetContext, “Caption={*Print}”, “” PushButton Click, “Text=Cancel” Window SetContext, “Caption={*Notepad}”, “” MenuSelect “File->Exit” End Sub 2.3. Note 2.3.1. Make sure you create a text file and use the correct path in the script. 2.3.2. For trial purposes, the script clicks on the ‘Cancel’ button, instead of the ‘Print’ button. 3. Executing the scripts To execute the script, click on the ‘Go’ button on the toolbar (14 th from the left; has a play icon and horizontal lines on it). You may even use the shortcut ‘F5’. Executing the script also compiles it if it hasn’t been done before. However, if you want to just compile your script, you may do so by clicking on the ‘Compile’ button (13 th from the left; has a green tick mark on it) or by using the shortcut ‘Ctrl+F7’. T he compile results (warnings and errors, if any) are displayed in the build tab a t the bottom of the screen. Once the script completes execution, the test manager window opens up to display the log i.e. the test results (pass/fail). Close the log file before executing the script again. To stop script execution at any point, use the shortcut ‘F11’. Reminder: The assignment is due on April 11, 2005. Check TA5 document for submission details. The coding phase begins today. Make sure you are clear with the basics before the end of class.

PAGE 92

83 Appendix G: Survey Informed Consent Social and Behavioral Sciences University of South Florida Information for People Who Take Part in Research St udies The following information is being presented to help you decide whether or not you want to take part in a minimal risk research study. Please read this carefully. I f you do not understand anything, ask the person in charge of the study. Title of Study: Test Automation ROI Metric Principal Investigator: Naveen Jayachandran Study Location(s): You are being asked to participate because you are enrolled in the course CIS/ISM 6930. The research study is based on one of the assignments you will be completing as part of regular coursework. 1.1 General Information about the Research Study The purpose of this research study is to develop a test automation ROI (Return On Investment) metric. For this, a test automation project (Assignment 5) comple ted by you as part of your routine coursework will be monitored and studied. In addition, the data provided by you via a survey after completing the assignment will be used for re search purposes. 1.2 Plan of Study You will be doing your coursework (Assignment 5) as usual. At the end of your assignment you will be asked to complete a survey form based on your experience s. This data will be used for the research study. 1.3 Payment for Participation You will not be paid for your participation in this study. 1.4 Benefits of Being a Part of this Research Study By taking part in this research study, you should learn the effective and appropri ate use of test automation tools and manual testing methods. It will also provide you with a sound basis to judge the testing methodology you would like to adopt for future projects.

PAGE 93

84 Appendix G: (Continued) 1.5 Risks of Being a Part of this Research Study There are no known risks incurred by being a part of this study. 1.6 Confidentiality of Your Records Your privacy and research records will be kept confidential to the e xtent of the law. Authorized research personnel, employees of the Department of Healt h and Human Services, the USF Institutional Review Board, its staff, and other individuals acting on behalf of USF, may inspect the records from this research project. The results of this study may be published. However, the data obtai ned from you will be combined with data from others in the publication. The published results w ill not include your name or any other information that would personally identify y ou in any way. Only the principal investigator and the members of this research study will have access to the data. The data will be stored at a secure location. 1.7 Volunteering to Be Part of this Research Study Your decision to participate in this research study is completely voluntary. Y ou are free to participate in this research study or to withdraw at any time. There will be no grade penalty or loss of benefits you are entitled to receive, if you stop taking part i n the study. 1.8 Questions and Contacts If you have any questions about this research study, you may contact Naveen Jayachandran at naveenjc@yahoo.com ; (813) 298 9504 or Dr. Dewey Rundus at rundus@csee.usf.edu or Dr. Alan Hevner at ahevner@coba.usf.edu If you have questions about your rights as a person who is taking part in a research study, you may contact the Division of Research Compliance of the University of South Florida at (813) 974-5638.

PAGE 94

85 Appendix G: (Continued) ISM 6930 – SOFTWARE TESTING SEMINAR SPRING SEMESTER 2005 TESTING ASSIGNMENT 5 Learning Experience Survey Age: Sex (M/F): Group (G1/G2): Sub Assignments (M1,A2/M2,A1) Section A 1. Do you have prior programming experience? If yes please specify (In ye ars and months). Ans. 2. Do you have prior software testing experience? If yes please specify (In years and months). Ans. 3. Have you taken a software testing course before the current course? If yes please specify the course and when and where you took it. Ans. 4. Do you have prior experience in the field of software test automation? If yes please specify (In years and months) Ans. 5. Have you used Rational Robot before? (Yes/No) a. If yes, describe your experience. Ans.

PAGE 95

86 Appendix G: (Continued) 6. Have you used any other test automation tool (E.g. SilkTest, WinRunner, etc.)? If yes, please specify the tool and describe your experience with it Ans. 7. How would you rate the effectiveness of test automation on a scale from 1 (poor) to 5 (excellent)? Ans. 8. How would you rate your learning experience with Rational Robot on a scale from 1 (difficult) to 5 (easy)? Ans. 9. How would you rate Rational Robot on usability on a scale from 1 (very easy to use) to 5 (very difficult to use? Ans. 10. Do you think test automation is more productive than manual testing? (Please explain why or why not?) Ans. 11. How difficult did you find writing test scripts on a scale from 1 (very easy) to 5 (very difficult)? Ans. 12. How many defects did you find during automated testing? Ans. 13. How many defects did you find during manual testing? Ans. 14. Did automation help you conduct consistent repeatable testing? (Yes/No) Ans.

PAGE 96

87 Appendix G: (Continued) 15. Did automation help you find regression bugs? (Yes/No) Ans. 16. Did automation help you run more tests? (Yes/No) Ans. 17. Did automation help you test more thoroughly? (Yes/No) Ans. 18. Did automation help you increase your confidence in the software? (Yes/No) Ans. 19. Did automation help you complete your testing faster? (Yes/No) Ans. Section B 1. How many person hours did it take you to learn the test automation tool sufficiently to do the assignment? Ans. 2. How many person hours did it take you to write the required test scripts for the assignment? Ans. 3. For a smaller project, would you recommend manual or automated testing? Why? Ans. 4. For a larger project, would you recommend manual or automated testing? Why? Ans.

PAGE 97

88 Appendix G: (Continued) 5. How big would a project need to be to use automated testing in a cost effective way? Ans. Section C 1. How many LLOC (Logical Lines Of Code) did you write in your test scr ipts? (Consider multi-line IF statements to be a single LLOC, etc.) Ans. 2. How many individual test cases did you create in automated testing? Ans. 3. How many individual test cases did you create in manual testing? Ans. 4. How many times did you run the automation scripts? Ans. 5. Did automation help you reduce your testing effort? (Yes/No) Ans. 6. How much effort did you spend to document your test cases on a scale of 1 (low) to 5 (high)? Ans. Section D 1. What could have made the learning experience better? Ans.

PAGE 98

89 Appendix G: (Continued) 2. How much do you think prior software testing experience affected your performance on a scale of 1 (little) to 5 (greatly)? If you had no prior testing experience, answer ‘N/A’. Ans. 3. How much do you think prior programming experience affected your performance on a scale of 1 (little) to 5 (greatly)? If you had no prior programming experience, answer ‘N/A’. Ans. 4. How much do you think prior automated testing tool experience affected your performance on a scale of 1 (little) to 5 (greatly)? If you had no prior programming experience, answer ‘N/A’. Ans. 5. How much did you learn about automated testing tools in this assignment on a scale from 1 (very little) to 5 (a great deal)? Ans. 6. Would you recommend that this assignment be performed by individuals or small groups? Why? Ans. 7. What recommendations do you have to improve this assignment? Ans.

PAGE 99

90 About the Author The author is a Certified Software Quality Analyst (CSQA) and a Maste r’s student at the University of South Florida with research interests in the field of software engineering, software testing, test automation and quality assurance (QA ). The Author has prior industry experience in software testing and QA. Currently, as a Gradua te Assistant, he tutors graduate students in software testing and test automati on tools. Updated information on his current research activities and contact details can be obt ained at www.smartsoftwaretesting.com