вход по аккаунту



код для вставкиСкачать
An Approach to Automatic Testing Exception Handling ∗
Shujuan Jiang1,2
Yongping Zhang 1 Dashun Yan1 Yuanpeng Jiang1
School of Computer Science & Technology, China University of Mining & Technology, 221008, China
Department of Computer Science & Engineering, Southeast University, 210096, China
50% of the operational failures of a telephone
switching system were due to faults in exception
handling and recovery algorithms. The Ariane 5 launch
vehicle was lost due to an un-handling exception
destroying $400 million of scientific payload [9].
Exception code is responsible for the detection
and handling of system conditions that could
potentially lead to failure. It is therefore important that
this code is tested effectively. The test period is the
most important period in the software life and its cost
is the biggest. In the study of Sinha and Harrold [10],
they observed that 23.3% and 24.5% of the classes
contained try and throw statements. If the efficiency
of testing exception handling is improved, then it will
improve the efficiency of the testing the whole system.
Because of the particularity of exception handling
code, it is difficult to test them by usual methods.
Bieman and horhuus tested exception handling by fault
injection [11,12]. When they inserted the fault,
however, they didn’t consider the special structure of
exception handling. From our previous analysis-based
work that had been used to analyze exception
propagation path and program structural testing, we
found that the structure of exception handling not only
maintainability, but also provides convenience for it’s
testing [13,14]. We test exception handing through
combining the structure of exception handling with an
“assertion violation” technique for injecting software
The paper presents an approach for automatically
testing exception handling. The aim is to provide
automated support for the testing of exception handing.
It also gives the framework of a tool that supports the
The rest of this paper is organized as follows.
Section 2 analyzes the problem of the testing exception
handling. Fault injection is introduced simply in
section 3. Section 4 presents our improved fault
injection method. Section 5 describes the basic
Exception handling mechanism can increase the
reliability of the system and improve the robustness of
the software. But the exception handling code that is
used to respond to exceptional conditions tends to be
the source of the systems failure. It is difficult to test
the exception handling by usual methods. This paper
proposes an approach to automatic testing of exception
handling based on analyzing the problems encountered
during testing exception handling. The method of
testing exception handling is efficient through
combining the structure of exception handling with
“assertion violation”. It also gives the supporting tool
that has been designed for the method.
Exception handling, software faults injection, software
testing, program mutation.
1. Introduction
Modern object-oriented software is inherently complex
and has to cope with an increasing number of
exceptional conditions to meet the system’s reliability
and robustness requirements. An exception handling
mechanism [1] provides support for detecting and
recovering from failure, and structuring fault-tolerant
activities in a system [2,3]. In some systems, the code
developed for error detection and handling is usually
numerous and complex. As a consequence, up to twothirds of a program can be for error handling [4]. The
exception handling mechanism is usually considered an
essential part of any modern programming language
(e.g. C++[5], Delphi[6], Java[7], etc.).
Although the purpose of the exception handling
code is to improve the robustness of the software,
people noticed that the exception handling code
contains more errors than the other parts of the
software. Indeed in a case-study by Toy [8] more than
This work was supported in part by the Opening Foundation of Jiangsu Key Laboratory of Computer Information Processing Technology in
Soochow University, Foundation of China University of Mining and Technology, Youth Foundation of China University of Mining and
Vol. 40(8), Aug 2005
In order to test exception handling, we must find a
set of test-data x∈ D, which raised the exception, and
test the exception handler.
Firstly, consider the case of a user-defined
exception. User-defined exceptions must be explicitly
raised in C++ by executing the throw statement.
Assuming the throw statement is contained in basic
block ni ∈ N, the problem of testing the raising of a
user-defined exception reduces to one of selecting testdata x ∈ D that will cause ni to be executed.
In the case of predefined exceptions the problem
is slightly different. Predefined exceptions are raised
when the language rules are violated at run-time and in
response to hardware errors. Test-data alone cannot test
the raising of exceptions in response to hardware errors.
In this paper, we focus on testing exception that
violates run-time language rules.
The input domain of most of programs, D, is
likely to be very large, but the input domain which can
causes an exception is likely to be small. It is very
difficult to find the test-data that can raise an exception
in so large input domain.
In fact, the testing exception handling mainly
focuses on two aspects. Whether can the exception be
thrown when the system occurs the exception?
Whether can the exception handler correctly respond to
the raised exception?
We test exception handling through combining the
structure of exception handling with an “assertion
violation” technique for injecting software faults. The
method can test the two aspects described above
completely and quickly.
constructs of the fault injection tool AutoETool and
section 6 draws conclusions.
2. Problem description
The most commonly accepted definition of exception
is the union of “error,” “exceptional case,” “rare
situation,” and “unusual event” [3]. The entity that is
raising an exception stops and waits for the completion
of the exception processing. The exceptions are usually
divided into two types: predefined and user-defined
exceptions [1]. The Predefined exceptions are declared
implicitly and are raised when the language rules are
violated at run-time and in response to hardware errors;
they can be raised implicitly during program execution.
The User-defined exceptions are defined and detected
at the application level; they can be raised explicitly in
the application via the raise statements. Exception
handling is the immediate response and consequent
action taken to handle the exceptions. An exception
handler is the code attached to an entity for one or
several exceptions and is executed when any of these
exceptions occur within the entity. An exception that is
not caught in a function is implicitly propagated to the
calling method. The use of exception is a powerful
mechanism that separates functional code from the
error handling code and allows a clean path for error
propagation. It facilitates the development of
applications that are robust and dependable by design.
A control flow-graph is a directed graph that
represents the control structure of a program. It can be
described as follows[9]: G = (N, E, s,e), where N is a
set of nodes, E is a set of edges of the form (ni, nj) and
s and e are unique entry and exit nodes such that s,e∈
N. A node, n∈ N, is a sequence of statements such that
if any one statement of the block is executed, then all
are executed. This is also known as a basic-block. An
edge (ni, nj) ∈ E represents a possible transfer of
control from the basic block ni to the basic nj. For
branch instructions the edges are associated with a
branch predicate.
A program is driven down a path in the control
flow graph by the values of its input variables and the
global state. These can be described as the vector
x = (x1, x2, … , xn). Each variable will have an
associated domain, Di, which can be determined from
the variable’s type. The total input space can be
defined as the cross-product of each of these domains,
D = D1 × D2 ×… × Dn. This allows a program input to
3. Software fault injection
Fault injection had been proposed for use in mutation
testing primarily as a mechanism for evaluating the
adequacy of test data [15,16]. Mutation testing injects
faults by modifying program text. Mutation testing can
require the creation of a vast number of mutant
programs. A program with N variables references can
have N2 mutant versions, and each program must be
recompiled and then tested. This process can be too
labor and time intensive for practical use in the general
testing of commercial software.
The mutation consists of weak mutation and
strong mutation. Weak mutation testing is less rigorous
but more efficient than strong mutation testing [11],
using weak mutation testing, mutant and original
program states are compared soon after the mutation is
executed, rather than after entire programs are run.
Weak mutation testing is much less expensive than
strong mutation testing, while almost as effective [17].
be defined as x ∈ D.
Vol. 40(8), Aug 2005
We can derive several possible faults to inject by the if
statement. The injection is a textual insertion at the
beginning of the function. For example, the injection
code might be the following (Using C++ syntax):
In this paper, we use weak mutation to inject fault.
The advantage of using fault injection to test
exception handling is that it can accelerate the program
to be failure. Testing exception handling by waiting for
the occurrence of failures in a system in the field is too
time consuming and not desirable. By intentionally
inserting faults to invoke the exception handing
capability, one can achieve more thorough testing in a
controlled environment and within a desirable time
case 1: i=-1; break;
case 2: i=isize+1; break;
case 3: j=-3; break;
case 4: j=jsize; break;
·Traditional fault injection
Example. Suppose there are the statements in the
source program as follows.
if (size = = 0)
throw new EmptyStackException();
4. The improved software fault injection
Pre-condition and post-condition contain logical
assertions or invariants that specify the expected
behavior of a program. These conditions are usually
not explicitly expressed in the program text, but rather
implicitly assumed [11].
In order to test exception handling, we improved
the traditional fault injection by combining “assertion
violation” with the special structure of the exception
handling. This section explains our choices and
presents the method of fault injection.
Exception raised is the notification of an
exception’s occurrence. The exception detection and
raising for predefined exceptions and user-defined
exceptions are different. For predefined exceptions,
detection and notification are usually performed
implicitly by the run-time system (either hardware or
software). For user-defined exceptions, the user must
explicitly test the exception conditions and raise the
For the example, the injection code might be the
case 5: size =0; break;
case 6: … break;
A fault injection tool according to a mutation rule
database can automatically generate the above blocks
of code. When injection is off, the injection status
variable is set to zero. When injection is on, the
injection status variable is set randomly to a value from
1 to 5. The number of injections and the probability of
choosing a particular injection can be determined by
the user.
4.2. Fault injection for the implicitly raised
4.1. Fault injection for the explicitly raised
The Predefined exceptions are declared implicitly and
are raised when the language rules are violated at runtime and in response to hardware errors.
For the exception that raised when the language
rules are violated at run-time, we can use the same
method as that used for the explicitly raised exception
to inject fault.
Example. Suppose there are the statements in the
source program as follows.
The explicitly raised exceptions can be raised explicitly
in the application via the throw statements. We can
gain the exception handling structure information from
source program easily. Then we can adopt two
different methods for fault injection due to the
exception conditions.
·Assertion violation fault injection
Example. Because the C++ compiler doesn’t check
whether the index of the array is legal, the developer
must check it by himself. Suppose there are the
statements in the source program as follows.
Where a, b and c are variables and sqrt is a standard
function. The pre-condition of the statement is a≠ 0.
We can obtain the fault injection code as following
from the assertion violation accordingly:
if ((i>=0 && i<isize) && (j>=0 && j<jsize))
return a[i][j];
else throw ArrayIndexException();
case 7: a =0; break;
case 8: … break;
Where ArrayIndexException is a user-defined
exception, isize and jsize denote the index of the array.
Vol. 40(8), Aug 2005
The default handler is that catches any type of
exception. It is put at the end of the list of handlers to
avoid pre-empting any that follow it.
For example, in C++, the default handler is
For those un-modifiable functions that exception
handling depends on, we can mutate the program using
the following way.
Example. Suppose there are the statements in the
source program as follows.
catch (…) {……}
Because there is not the type of exception in the
default handler, we must analyze the types of throwing
exception and the types of exception handler in the
guarded block. Then we can select one of exception
types that are not among the types of exception handler.
And then using throw statement explicitly raises this
exception within the guarded block.
Example. Suppose there are the statements in the
source program as follows.
p = new(sizeof(int));
if (p = = NULL) throw bad_alloc();
Where new() is a library routine and we may have
limited access to its source code. We can automatically
mutate the if condition into something like:
if (p= =NULL)||(injection= =12)
throw bad_alloc();
When the value of injection is 12, the exception
handler for “bad_alloc” can be tested.
For testing exception handlers that are in response
to hardware errors, it is very difficult if we use
traditional fault injection way [12]. If the system occurs
the hardware error, It is also difficult to recover by
software way. It generally performs some work that
saves some information before terminating program.
For this case, it is enough that we only test the
exception handler. We use a simple and efficient way
as follows. We can know the type of the handler from
the exception handling information that gained from
the source program. Then we add throw statement to
the guard block. The problem of testing this kind of
exception reduces to testing the explicitly raised
try {
// guarded block
} catch(type1 id1) {
// handle exceptions of type1
} catch(type2 id2) {
// handle exceptions of type2
} catch(...) {
// default handler
We can automatically derive a throw statement to
inject into the try block. Such as:
throw Except3();
Where the exception type, Except3, is neither type1and
type2, nor the sub-type of them.
4.3. Fault injection for the default handler
Interface of fault injection
take out
5. The fault injection tool
by tree
run and
the results
fault injection
Fig. 1. The system architecture of AutoETool
software probes placement and the content of fault
injection. The aim is to provide automated support for the
testing of exceptions. The system architecture of the
AutoETool is shown in Figure 1.
As in Figure 1,the tool consists of four subsystems:
Testing exception handling is a complicated and hard task,
especially in large software systems. In order to help
developers address the kinds of problems described in the
previous section, we have designed a fault injection tool,
called AutoETool, which can assist developers in the
Vol. 40(8), Aug 2005
(1) Take out exception handling information. By
scanning the program source code, it is easy to extract the
exception handling information, such as try block, catch
block, throw statement, the name of class that contains
exception handling, the name of function that contains
exception handling. Then it displays the information on
the left side of the “interface of fault injection” with tree
structure. In this subsystem, it also analyzes the type of
throwing exception, the type of exception handler, and
the relationship between them. It can find the exception
handler that is never executed based on the information.
(2) Interface of fault injection. It displays the
exception handling information by tree structure on the
left side of the window. On the right side of the window,
it displays the source program of the tested function or
class that is chosen on the left side of the window.
If you change the choice on the left side of the window,
the content of the right side of the window will change.
(3) Fault injection. According to the chosen exception
on the left side of the window, the developers can find the
appropriate place to inject fault. If you choose a throw
statement, it can generate a set of injection code from the
mutation rule database accordingly. You may choose the
generated code directly or modify it to meet the fault
injection requirements. Then inject the fault to the source
program on the right side of the window.
The database of mutation rule contains assertion rules,
which provide assistance for mutation code. The
developers can design mutation code based on the
mutation rule.
(4) Run and compare the result. Run the mutated
program and determine whether the exception handler is
correct from the output of the fault injection program.
We would like to thank Baowen Xu, Xiaoyu Zhou,
Changhai Nie,Yuan Liu, Liang Shi and other people at
Ada lab who met to discuss the related issues for their
valuable comments.
[1] J.B. Goodenough, Exception handling: issues and a
proposed notation, Communications of the ACM 18,
12, 1975, 683-696.
[2] A.F. Garcia, D.M. Beder, C.M.F. Rubira, An
exception handling software architecture for
developing fault-tolerant software, Proceedings of
the 5th IEEE High Assurance Systems Engineering
Symposium, USA, 2000, 311-320.
[3] J. Lang, D.B. Stewart, A study of the applicability of
component-based real-time software technology.
ACM Transactions on Programming Languages and
Systems, 20(2), 1998, 274-301.
[4] A.F. Garcia, C.M.F. Rubira, A. Romanovsky, and et
al, A comparative study of exception handling
mechanisms for building dependable object-oriented
software, The Journal of Systems and Software,
2001, 59 : 197-222.
[6] .
[7] J. Gosling, B. Joy, G. Steele, G. Bracha, The Java
Language Specification (2nd Ed.)
[8]W.N. Toy. Fault-tolerant design of local ESS
processors. The Theory and Practice of Reliable
System Design, 1981.
[9] N. Tracey, J. Clark, K. Mander, J. McDermid.
Automated test-data generation for exception
conditions. Software-Practice and Experience, 2000;
[10]S. Sinha, M.J. Harrold. Analysis and testing of
programs with exception-handling constructs. IEEE
Transactions on Software Engineering 26,
September 2000.
[11] J.M. Bieman, D. Dreilinger, Li-jun Lin. Using fault
injection to increase software test coverage[A].
Software Reliability Engineering Proceedings Seven
International Symposium[C]. New York: IEEE
Computer Society Press,1996, 166-174.
[12] R. horhuus. Software Fault Injection Testing(Master
Thesis). Stockholm, February 2000.
[13] S.J.Jiang, B.W.Xu.Exception handling—an approach
to improving software robustness. Computer Science
(in Chinese), 30(9), 2003, 169-172.
[14] S.J. Jiang, B.W. Xu, L. Shi. An approach to analysis
International Conference On Software Engineering
And Applications. November 9-11, 2004. MIT
Cambridge, MA, USA,300-305.
[15] T. A. Budd. Mutation analysis: Ideas, examples,
6. Summary
Exception handling mechanism is an important feature of
modern programming languages. It offers a strong
support for reliable software systems. But the exception
handling code is more prone to error than the other parts
of the system. This paper proposes an approach to
automatic testing exception handling through combining
the structure of exception handling with an “assertion
violation” technique for injecting software faults. We
have developed a tool, called AutoETool, to support the
fault injection. It can provide an automated support for
the developers in the placement and the content of fault
The mutation rule database is incomplete and needs
improved. This is also our next research work.
Vol. 40(8), Aug 2005
problems and prospects. In B. Chandrasekaran and S.
Radicchi, editors, Computer Program Testing, pages
129–134. North-Holland, 1981.
[16] R. A. DeMillo, D. S. Guindi, W. M. McCracken, A.
J. Offut, and K. N. King. An extended overview of
the mothra software testing environment. Proc. ACM
SIGSOFT/IEEE Second Workshop on Software
Testing, Verification, and Analysis, pages 142–151,
July 1988.
[17] A. Offutt, S. Lee. An empirical evaluation of weak
mutation. IEEE Trans. Software Engineering,
20(5):337–345, May 1994.
Vol. 40(8), Aug 2005
Без категории
Размер файла
899 Кб
1089858, 1089851
Пожаловаться на содержимое документа