Nickolay Platonov
25 July 2013

Code coverage with Siesta 2.0

Siesta 2.0 was released last week during SenchaCon 2013 in Florida. The main features of this release are the brand […]

Siesta 2.0 was released last week during SenchaCon 2013 in Florida. The main features of this release are the brand new UI (based on Ext JS 4.2) and the ability to generate code coverage reports. Code coverage is a big and exciting addition that was frequently requested in the past year, and we already use it ourselves internally. Since we think there might a few additional features left to be implemented to further improve the coverage support, we’ve marked the coverage feature as “experimental” for another couple of releases. We reserve the right to slightly change the API if required, strictly based on your feedback. Please give it a try and let us know what you think, and how we can improve it even further!

What is code coverage?

Code coverage is a metric, measuring what parts of your codebase are exercised when running your test suite. In other words it highlights code that has never been executed by the test suite and therefore should be treated as unsafe and a potential source of bugs.

At the same time, code coverage by itself does not guarantee that your code actually works. “Covered” code can still produce wrong results, such as logical or algorithmic errors. The only real assumption you can make about covered code is that it won’t throw unexpected exceptions. The main point of interest is actually the absense of coverage. That is a much more valuable piece of information, as it tells you where you should focus your testing efforts and write additional unit tests. If a core class in your model layer is not exercised by any test, it should be a clear call to action!

How does it work?

To be able to collect code coverage data, your codebase first has to be instrumented. This means it’s transformed, and additional code statements are injected that count which statements are actually executed. Siesta is using the awesome Istanbul library for this purpose.

For example, if your file contains only one simple assignment:

var a = 1;

After being instrumented, it will look like this:

if (typeof __coverage__ === 'undefined') { __coverage__ = {}; }
if (!__coverage__['filename.js']) {
__coverage__['filename.js'] = {"path":"filename.js","s":{"1":0},"b":{},"f":{},"fnMap":{},"statementMap":{"1":{"start":{"line":1,"column":0},"end":{"line":1,"column":10}}},"branchMap":{}};
}
var __cov_1 = __coverage__['filename.js'];

__cov_1.s['1']++;var a=1;

As you can see the amount of code increases and naturally instrumented code will take longer to execute. But don’t worry, the instrumentation is unobtrusive and does not change the semantic of your code.

Enabling code coverage in Siesta

To enable the code coverage module, simply set the enableCodeCoverage : true config option in your harness:

Harness.configure({
    enableCodeCoverage      : true
})

Additionally, you need to include the “siesta-coverage-all.js” file on your harness page after the regular Siesta JS file:

<script type="text/javascript" src="../siesta-coverage-all.js"></script>

You can find a sample setup in the “/examples/code_coverage.html” file inside the Siesta package.

Siesta will instrument only files marked explicitly in the `preload` config. To tell Siesta to instrument a JS file, provide a instrument : true config in the file’s preload descriptor:

    Harness.configure({
        preload         : [
            {
                type        : 'js',  // optional if file ends with ".js"
                url         : 'some_file.js',
                instrument  : true
            }
        ],
        ...
    })

Additionally when using the Ext JS testing layer, Siesta will install an on-the-fly instrumentation hook into the `Ext.Loader` code.

Coverage per class

Normally code coverage information is collected on a per-file basis. However, in web development it’s common to concatenate several JavaScript files (each containing one class) into one big bundle file. Getting file based coverage for such a “bundle” file will not be very useful, as it can be huge and if you want to check the coverage for one specific class in the bundle you will have to search and scroll a lot.

For the Ext JS testing layer, Siesta can collect coverage information on a per class basis. In this mode Siesta looks for Ext.define() calls in the source files, and instruments only the detected classes (which is normally what you want). The resulting report in this mode will contain information about Ext JS classes, not files. As you can see in the image below, when clicking a covered class you can immediately see any source code missing coverage (highlighted in red).

Reports and Continuous Integration

Code coverage metrics should naturally be part of your Continuous Integration process. You can easily generate code coverage reports using our Phantom JS or WebDriver launcher scripts. To generate a coverage report, simply add --coverage-report-format when launching your test suite. Recognized values are `html`, `lcov` or `raw`.

Conclusion

Code coverage information is an important metric measuring one aspect of your test suite health. It indicates clearly where additional unit tests are required. The core classes of your application should definitely have a high level of code coverage.

We’re hoping Siesta will make it really easy for you to gather coverage information. Please give the Siesta 2.0 release a try and let us know how it works for you!

Nickolay Platonov

Siesta Testing