Challenges in Software Testing

 

David S. Rosenblum
Professor of Software Systems 
University College London

 

Software testing is one of the most important and widespread methods used by software developers to validate the behaviour of a software system prior to its release to customers and users.  It is used to validate both functional correctness and many non-functional properties, and it is used to estimate the reliability (i.e., likelihood of failure, or mean time to failure) of a software system.

Software testing has seen a number of recent innovations being adopted by developers.  Among these innovations is test-driven development as advocated for agile development (such as Extreme Programming), in which test cases are continually developed in tandem with the development of source code.  There are also model-based testing approaches that exploit design models expressed in UML and the Model-Driven Architecture (MDA).  And after a software system has been released, development organisations increasingly perform testing driven by the use of field data disseminated to testers from the deployed versions of software systems.

Despite these innovations and advances, dramatic changes have been taking place and continue to take place in the ways in which software systems are developed and deployed, and these changes make software testing a much more difficult endeavour than it was for traditional monolithic system development.  In particular, a number of challenging testing issues arise in testing component-based software systems, service-oriented software architectures, and context-aware pervasive computing systems.

Component-based software systems are built from ¡°off-the-shelf¡± components developed independently by third parties.  Apart from open-source components (whose source code is made publicly available), such components must be treated by software integration engineers as black boxes, posing engineers with a lack of access to development artefacts traditionally used in testing (particularly source code) and to techniques for exploiting those artefacts (such as source-level instrumentation, and flow analysis to support white-box coverage testing).  One approach to dealing with this problem is to perform black-box testing of a component using statistical analysis of the correlation between the component¡¯s behaviour and the corresponding portions of its input domain that are exercised to produce the behaviour.  Another approach involves packaging components with specialised metadata that expose key internal properties of a component that are relevant to testing; such metadata potentially provide the integration engineer with just enough information about component internals to support software testing, but not enough to reveal trade secrets or other proprietary information of the component vendor.

Service-oriented software architectures (such as Web services) provide platforms for dynamic discovery and invocation of heterogeneous components deployed in a distributed, networked manner as reusable services having well-defined interfaces.  Services potentially provide functionality to several applications simultaneously.  Not only are services like other components in that they are developed independently by third parties, but they are also deployed independently.  Thus, integration engineers wanting to develop service-oriented applications face the problems mentioned above regarding off-the-shelf components, plus the additional problem of a lack of control over the integrated collection of services, making it infeasible to employ intrusive and disruptive testing activities, such as load testing, that may interfere with the use of the services by other applications.

Context-aware pervasive computing applications are typically deployed on small computing devices such as wireless handhelds and are designed to react to continual changes to system context (i.e., attributes of the system execution environment).  Context may be internal (such as the device¡¯s available memory, battery level, CPU, wireless transmitter power, etc.) or external (the device¡¯s geophysical location, attributes of the location such as a room number or set of room facilities, the device¡¯s set of nearby neighbour devices, etc.).  Context may be sensed by a system in synchronous fashion (such as querying for the set of neighbour devices prior to selecting a neighbour device for communication) or in asynchronous fashion (such as a program exception being raised to indicate low battery level).  And context may be extensively interpreted by a middleware layer that maps collections of low-level context information into high-level context attributes of interest to a particular application.  Context is thus essentially an additional input space of the system, but one that spans not just different executions of the system but also several layers of software.  In addition, this input space contains not just the initial inputs of the system, but rather is continuously fed to the system.  This multi-dimensional nature of context precludes the use of traditional approaches to test case selection, in which test suites are formed as sets of carefully chosen initial input vectors, because such approaches can lead to an exponential blowup in the number of test cases needed for effective testing of the system.

The new modes of development and deployment discussed above have the potential to produce dramatic gains in application scalability and development productivity.  Yet they also produce enormous challenges for testing, in large part because traditional approaches to testing appear to have limited applicability for these new kinds of systems.  Thus, fundamentally new approaches to software testing are needed that account for the peculiar characteristics of these new kinds of software systems.