Today we are glad to announce the Siesta 5.1.0 release. In this release, we present a completely new, re-engineered code coverage module. It might seems strange to release 5.1 immediately after 5.0, but this is what the semantic versioning rules dictate, so we follow.
Code coverage in previous Siesta versions
To achieve the goals mentioned above, we had to heavily patch Istanbul. This of course made it very hard to upgrade Istanbul to newer versions. We also had to patch the Istanbul reports – so we ended up supporting only some of them. Because of this, Siesta’s code coverage module got stuck, it had some bugs which were hard to fix and it was not possible for us to adopt the new ES6 syntax, like arrow functions which is now supported by all modern browsers.
Since version 5, Siesta also started to support testing Node.js code, and code instrumentation in the browser just was not general enough to support both browsers and Node. So we decided that this module should be rewritten from scratch.
The new code coverage in Siesta 5.1.0
With the new code coverage module we just provide a thin integration layer on top of Istanbul and we do not patch its code. Our goals with the re-implementation were:
- Easy upgrades to new versions of Istanbul to adopt new features
- Ability to generate all reports supported by Istanbul.
To achieve these goals, the code instrumentation and reports generation have to happen in Node. As a result, code coverage is now available only from the command line and not in the web interface and we found this to be a reasonable trade-off.
The code coverage in Siesta 5.1.0 is controlled by the
--nyc.* command line option group. Such options, like
--nyc.reporter are translated directly to the nyc (Istanbul’s command line interface tool), so they work in the exact the same way as in
To collect code coverage information and generate a report, you should specify the
--nyc.reporter option (old name
--coverage-report-format is still supported as an alias). All Istanbul reports listed here are supported.
For example, to get a text coverage report for Node.js code, simply run:
> bin/nodejs examples/nodejs/code-coverage --nyc.reporter=text
Or, to generate an HTML coverage report for browser JS code:
> bin/webdriver examples/sencha-extjs --nyc.include='**/5.code_coverage' --nyc.reporter=html --include code_coverage
The code coverage module works with cloud testing providers like Sauce Labs too. The only detail is that the Sauce Labs tunnel should be configured with additional options, so one should allow the Siesta launcher to establish it:
> bin/webdriver examples/sencha-extjs --nyc.include='**/5.code_coverage' --nyc.reporter=html --include code_coverage --saucelabs $SL_USER,$SL_KEY,$SL_DOMAIN
--nyc.exclude options. This proxy is then automatically set for all WebDriver sessions.
For the purposes of including/excluding, the URL of the request is transformed into a “virtual” file name. For example, the URL
/http/project.com/lib/app.js Then the content of the request is passed to Istanbul for instrumentation. We use Istanbul’s caching, meaning any repeating requests for the same file will be processed much faster (it will still take some time, since Istanbul calculates the hash of the file content).
When testing over https, which is becoming more and more popular, Siesta automatically generates self-signed certificates for all encountered hosts. IE11 may require additional setup because of this since its WebDriver version does not support the “acceptSslCerts” capability. Please see code coverage guide for more information.
Pre-instrumentation can be done using:
- babel-plugin-istanbul Please find more information in the nyc documentation
instrumentcommand of the
nyctool. More information can be found by running
npx nyc instrument --help(or
node node_modules/.bin/nyc instrument --help)
So what are the changes in the API? Let’s review the configuration options from the previous implementation and their equivalents in the new:
enableCodeCoverage– project option, removed. Code coverage is now always controlled from the command line with the
coverageUnit– project option, removed. There’s no “coverage unit” concept in the new implementation. It was supporting getting better reports from the bundled code. Same result can be achieved with the pre-instrumentation of the source code before bundling.
excludeCoverageUnit– project options, removed. Use
--previous-coverage-reportcommand line option has been removed. To create a combined coverage report from several consequent test suite launches, specify the
--nyc.clean=falsefor the 2nd and following launches
Integration with your CI dashboard
At Bryntum we use TeamCity as our Continous Integration tool and we now have our code coverage reports displayed with each build. This makes it easy to keep an eye on our coverage levels to make sure we don’t release under tested code. Below you can see our report for the Bryntum Grid inside our TeamCity dashboard: