Download presentation

Systematic testing of a software system builds underlies modern software development methodologies. Continuous Integration (CI), a widespread approach, is based on testing of each change made to the source code in addition to the traditional testing of nightly builds. Further development of this technique introduces the pre-commit testing: a set of essential tests that a change-set has to pass before it can arrive into the main source code repository. It helps to avoid situations when a serious bug in the software system blocks the development process in a whole team until the bug is fixed.

The development of complex software systems composed of many interacting components usually requires extensive pre-commit testing because operability of each component should be verified. Manual selection of tests for the essential testing is not very reliable because the way a change in a single component affects the software system may be completely unexpected to the developer. In practice this results in a slowdown of the development process because the duration of the pre-commit testing is one of the factors limiting the speed of working on the software project.

A number of approaches to reduction of the testing volume based on comparison between the source code changes and the source code coverage is described in literature and researched from the theoretical point of view. The main idea is to exclude the tests that do not cover the changed parts of the program from the testing. Deployment of these techniques into the development process of a complex software system may encounter a number of difficulties that are mostly caused by the size of the code coverage data. The most complicated part is often keeping this data up to date because the testing of an instrumented build is usually several times (3-5) longer than the regular testing. Thus a growing load on the testing system may outweigh the gain achieved by the optimization. Storing, transferring and timely analyzing the code coverage data may also cause difficulties because its’ size may come up to hundreds of gigabytes.

In this report we will present our experience in deploying this technology into the development process of an existing software product. Mentioned complications were overcome by lowering the detail level of the source code coverage. Instead of traditional basic blocks coverage we were using procedure-level coverage (classes, methods or modules may be used as well). This approach requires instrumenting only the entry point of each procedure. It may result in a significant (up to 80%) speed up of coverage data update. Instead of considering each test separately the logical groups of test were used. It reduces the amount of the coverage data and makes the tool operation more transparent and visual because it forms the testing of big blocks and each block has a clear purpose. General decrease in the number of details in the coverage data makes it more stable over time. This allows updating it periodically, not for the every change in the source code. Other practical aspects of this work will be covered such as developing the source code changes analysis tool and verifying its’ correct operation.

Alexey Salmin

Alexey Salmin

Alexey Salmin is working at Intel Corporation as a software engineer in the Intel Compiler QA group. He has an expertise in designing, developing and supporting continuous integration and automated testing systems. Alexey has a master degree in applied mathematics and informatics.

Alexander Stasenko

Alexander Stasenko

Alexander P. Stasenko is an Intel compiler QA engineer at Intel Corporation. Alexander’s area of interest includes automated test generators and automated test case reduction. Alexander has degree of candidate of science for work in area of compiler front-ends and functional languages.