A Study on the Efficiency of a Test Analysis Method Utilizing Test-Categories Based on AUT and Fault Knowledge. 15th Aug. 2014 ICACTCM’2014 Tsuyoshi Yumoto, Keiji Uetsuki, Tohru Matsuodani, Kazuhiko Tsuda
Agenda 1. INTRODUCTION 2. VARIABILITY OF TEST ANALYSIS RESULTS 3. AN APPROACH OF TEST CATEGORIES BASED TESTING 4. REMARKS FROM THE VERIFICATION EXPERIMENT 5. CONCLUSION 2
1. Introduction Along with a rapid increase in the size and complexity of software today, the number of required test cases is also increasing. 3 Testing Development ・Requirement ・Specification ・Design ・Code Development ・Requirement ・Specification ・Design ・Code increase in size increase complexity Testing 10 times for 30 years Quantity FP Test cases Quantity FP Test cases Test cases is equals the FP(function point) sum to the power of 1.3 to 1.5 (by Capers Jones
2. VARIABILITY OF TEST ANALYSIS RESULTS Generally, a significant number of testers should be assigned to a project in order to manage this increase of test cases. Therefore, there are no clearly defined general rules for test development process (see next page). • They are developed according to the individual’s own judgment. • This has the potential to cause the lacking or duplication of test cases. Test analysis results when rules are not defined and interpretation is left to the individual 4
2. VARIABILITY OF TEST ANALYSIS RESULTS(Cont.) Definition of Test development process and Test analysis • Testing performed at each level depicted in the V model has a process similar to the development process, and three activities, test analysis, test design, and test implementation in the test process are called Test development process. 5 A part of V-model Test analysis An activity of selecting and organizing items to be covered by the test .
3. AN APPROACH OF TEST CATEGORIES BASED TESTING A set of rules for a test analysis method for black box testing utilizing Test- Categories based on the Application Under Test (AUT) and fault knowledge have been proposed. Overview of the method The logical structure of a feature Test –Category The procedure and document format 6
3. AN APPROACH OF TEST CATEGORIES BASED TESTING The logical structure of a feature • It can be used to test the feature in a MECE way . Each box in the logical structure can be a useful guide to determine the required test conditions. 7 Logical Structure… a MECE way to find test conditions from features.
3. AN APPROACH OF TEST CATEGORIES BASED TESTING Test-Category • In order to have a consistent interpretation of determining test conditions, a 8 name specialized for the AUT is put in each box of the logical structure. • In order to ensure for clearly understanding the meaning of Test-Category , potential failures and/or faults which may arise for Test-Category are discussed. Knowledge of AUT Knowledge of faults experienced in the past Building consensus on the decided Test-Categories
3. AN APPROACH OF TEST CATEGORIES BASED TESTING The procedure and document format • Things used to determine test conditions from a test basis are defined. 9 The outputs are eliminated by layering elements based on the test case structure.
3. AN APPROACH OF TEST CATEGORIES BASED TESTING (Cont.) The Main benefit of this method • Higher test coverage overall, delivering higher quality testing – By implementing the set of rules it will be easier to determine the necessary 10 test conditions. Our hypothesis The following issues currently make determining the necessary test conditions difficult: 1. Certain aspects of specification are not written if they are thought to be obvious. 2. Specification is not completely written within single target section in a document. (for example: a behavior about a combination of functions) – When many testers are involved in test development and proceed according to the procedure and document format, all of the testers can carry out their work according to the same set of rules. The developed suite of test conditions are more comprehensive and do not contain duplicates
3. AN APPROACH OF TEST CATEGORIES BASED TESTING (Cont.) Another benefits of this method • High Readability – The procedure is based on the structure of the test conditions defined in the proposal of 11 this method . Elements which are included in the test basis are classified into specification-items, expected results, and test parameters. One-by-one, these are determined and selected through this procedure. Thus, the end result readability is improved. • Variation in the interpretation is minimized – Once a consensus regarding the building of Test-Categories is met, team members can more easily determine and select specification-items. • Repeatable – Since this method is structured, standardized and easy to walk though, testers will come up with consistent and repeatable test analysis.
4. REMARKS FROM THE VERIFICATION EXPERIMENT The verification experiment Overview Explain about Test category 12 Results of the exercise conducted Feature to be tested according to Test-Categories Test categories Spec items Expected results Test Basis Test Parameters Feature to be tested Spec Item Expected results Test Parameters Results were classified to test categories Results of the exercise conducted according to Test-Categories 1st exercise 2nd exercise Comparison and Evaluation for Spec items B - A A+
4. REMARKS FROM THE VERIFICATION EXPERIMENT (Cont.) Evaluation results of the verification experiment • Eight comparison results table were taken from the two verification experiments. • There was a measurable improvement resulting from implementing the Test- Categories method for seven out of eight teams. • However, there was no indication as to exactly which categories received the greatest benefit from implementing the method. Music reproduction equipment Flight Booking Application 13 Logical Team Structure Logical Structure Team TM1 TM2 TM3 TM4 TM5 TM6 TM1 TM2 Conv B A B B B B Conv A A Input Input A B Output - - - - - A+ Output A A Storage - A+ - A+ A+ - Storage A A Support B B B B B B Support B A Mngt B A A A+ A A+ Mngt B A
4. REMARKS FROM THE VERIFICATION EXPERIMENT (Cont.) Evaluation using patterns of input and output data of AUT • In order to classify and evaluate the results of the verification experiments, patterns of inputting and outputting data into/from the AUT (I/O data patterns) is focused as a comprehensive AUT model. The total combined patterns of inputting and outputting data into/from the AUT can be summarized into the nine patterns 14
4. REMARKS FROM THE VERIFICATION EXPERIMENT (Cont.) Evaluation results by I/O data patterns Music reproduction equipment Flight Booking Application 15 Logical Team Logical Team Structure Structure TM1 TM2 TM3 TM4 TM5 TM6 TM1 TM2 Conv P1 - - - - - - Input P1 B B Conv P1 B A+ B B B - Support P1 - A+ Support P1 B B B B B B Support P1 B B Mngt P1 B A+ A+ A+ A+ A+ Storage P2 A+ A+ Storage P2 - A+ - A+ A+ - Input P4 - - Conv P4 B B B B - B Mngt P4 B A Mngt P4 B B B A+ B A+ Output P4 B B Output P7 - - - - - A+ Output P4 A A Input P7 A - Conv P7 A A Output P7 B B Summarizing results in the I/O data pattern P2 is 100% for both the experiments. It can be said that P2 has had the most significant result. There were no detailed description about all spec items for P2 in the test basis. These results may suggest that the results are aligned with our assumption in Section 3. Music reproduction equipment Flight Application Evaluation level I/O Evaluation level pattern A+ A - B A+ A - B I/O pattern P1 6 0 7 11 35.3% P1 1 0 1 4 20.0% P2 3 0 3 0 100.0% P2 2 0 0 0 100.0% P4 2 0 1 9 18.2% P4 0 3 2 3 50.0% P7 1 0 5 0 100.0% P7 0 3 1 2 60.0% the persentage = (A+ + A) /(A+ + A + B)
Conclusion • Through these verification experiments it has been observed that after briefly 16 explaining this proposed method to participants there was a measurable improvement in quantity and consistency of spec-item which they were able to determine. • Moreover, by analyzing I/O data patterns, the patterns of inputting and outputting data into/from the AUT, a part of the evaluation results from the verification experiments aligned with the assumption in the proposed method. • Further verification experiments are necessary in order to carry out trend analysis with higher accuracy. • Conducting further experiments and deepening our understanding of the tendencies and factors relating to effectiveness of the proposed method, rules for creating Test-Categories based on the AUT knowledge and fault knowledge can be more refined.
References  C.Jones,"Estimating Software Costs 2nd Edition": McGraw-Hill, 2007.  D.Longstreet, "Productivity of Software from 1970 to Present", http://www.softwaremetrics.com/Articles/history.htm, 2000  C.Ebert, and C.Jones. "Embedded Software: Facts, Figures, and Future." IEEE Computer 42.4 ,2009, pp 42-52.  N.Owada"System integration 'straightforward way'",Nikkei Business Publications, Inc,2009 (In Japanese).  S. Eldh, H.Hansson, and S.Punnekkat, “Analysis of mistakes as a method to improve test case design.” In Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference on.IEEE,2011,pp. 70-79  T.Yumoto, T.Matsuodani, and K.Tsuda. "A Test Analysis Method for Black Box Testing Using AUT and Fault Knowledge.": Procedia Computer Science 22 ,2013, pp.551-560.  ISTQB FLWG, "Foundation Level Syllabus Version 2011": International Software Testing Qualifications Board , 2011.  T.J.Ostrand, and M.J.Balcer, "The category-partition method for specifying and generating fuctional tests.":Communications of the ACM 31.6 ,1988,pp. 676-686.  M.Grindal, and J.Offutt, "Input parameter modeling for combination strategies.": IASTED International Conference on Software Engineering (SE 2007), 2007.  G.J.Myers, C.Sandler, and T.Badgett, "The art of software testing ": Wiley, 2011. 18
References (cont.)  B.Beizer, "Software testing techniques ": Dreamtech Press, 2003.  Y.Nishi, "Viewpoint-based Test Architecture Design.": Software Security and Reliability Companion (SERE-C), 2012 IEEE Sixth International Conference on. IEEE (2012), pp.194- 197.  K.Akiyama, T.Takagi, and Z.Furukawa, "Development and Evaluation of HAYST Method Tool (Software Testing)." : SoMeT. ,2010, pp.398-414  S.Omura, "Phenomenology of general system": gohodoshoppan, 2005 (In Japanese).  "IEEE standard for software test documentation ": IEEE829. 2002,2002  Rasiel, E. M. The McKinsey Way. McGraw-Hill; 1999.  Tsuyoshi Yumoto,Tohru Matsuodani,Kazuhiko Tsuda. "A practical Using Method for Efficent Design of Functionai Testing": 75th National Convention of IPSJ; 2013 , paper#5B-4 (In Japanese).  J.A.Whittaker, "How to break software ": Addison Wesley, 2003. 19