Speeding Up Siesta – Sandboxing Now Optional
Sandboxing – what is it exactly..?
Every since the first Siesta release, tests have always been executed in their own iframes – “sandboxed”. This feature of Siesta assures that tests get a fresh context as they start and can’t interfere with other tests by changing global browser properties or defining global variables. Another big bonus is that you don’t need to manually clean up after your tests. As a sandboxed test is being started, this is roughly what is happening under the hood.
- 1. Siesta creates a new iframe and adds it to the page
- 2. The JS and CSS resources defined in your preloads are injected into the iframe
- 3. Siesta waits for the resources and the DOM to be ready and then executes the test code.
This is a pretty high price to pay, especially for pure unit tests which run fast. The setup is typically measured in seconds, whereas a unit test could finish in 100ms. For the worst case, this means a magnitude of 10-20x of setup time compared to test code execution time. Which of course tells us that there could be room for improvement for certain well structured tests.
Introducing the “sandbox” config option
In the latest nightly build (download here, a new config option has been added to Siesta called “sandbox” and it’s important enough to deserve a special blog post.
The `sandbox` option is true by default which keeps the old behavior intact, but you can now set it to false on either the Harness or on test groups. Disabling the sandbox feature takes a bit of thought, as it should not always be used. Disable sandboxing only when:
- 1. You are sure that your tests do not modify any global state
- 2. A group of tests share the same JS/CSS resources, i.e. tests don’t preload individual scripts
When not to use sandboxing
If your tests are creating global variables or in other ways modifying properties of the BOM, they are likely not suitable for sharing context with other tests. Let’s illustrate what we mean by “modifying global state”:
[crayon striped=”false” lang=”js” nums=”false” toolbar=”false”]
// 1st test file
// ============================
// “Helper” is a config object
Helper = { prop : ‘value’ }
// setting some global config before the test
MyApp.someConfig = “someValue”
// registering some store
var store = new Ext.data.Store({ storeId : ‘store’ })
// 2nd test file
// ============================
// “Helper” is an ExtJS model class
Ext.define(‘Helper’, { …. })
// FAIL: the “MyApp.someConfig” is expected to have a default value
// but it has been changed during execution of test 1
t.is(MyApp.someConfig, ‘defaultValue’, “`someConfig` has a default value”)
// The store with id `store` is kept from the 1st test
t.notOk(
Ext.data.StoreManager.get(‘store’),
“Should be no store with id `store`”
)[/crayon]
As you can see, without sandboxing, there can be conflicts in even the simplest code. Most of them are related to shared state – global variables and singletons. If you are careful with the shared state, and your tests rely fully on local variables within the main test function, you should be able to disable sandboxing and improve your test suite performance significantly.
Implementation
Let’s now see what will happen if the `sandbox` option is disabled for a group of tests. Siesta will collect all tests having this option disabled and split them into chunks. Every chunk will have exactly the same values for the test configs which influence the initial setup of the page: `preload`, `alsoPreload`, `hostPageUrl`, `requires` and `loaderPath`. Example:
[crayon striped=”false” lang=”js” nums=”false” toolbar=”false”]Harness.start(
{
group : ‘Group 1’,
sandbox : false,
preload : [ ‘file1.js’ ],
items : [
‘test1.t.js’, // test1 and test2 will be run in the same “sandbox”
‘test2.t.js’,
{
// this test will have a separate sandbox, since the `preload` option
// (which influence initial page setup) is different
url : ‘test3.t.js’,
preload : [ ‘file2.js’ ]
}
]
},
…
)
[/crayon]
The 1st test in every chunk will be run normally. Starting from the 2nd one, tests will skip the `isReady` check and `setup` methods. This is because all the setup is done for the 1st test. This behavior may change (or be made configurable) in the future.
Important: Before a test is launched in a “dirty” sandbox, a cleanup is performed. On the “raw” browser testing level, the cleanup process includes removal of all “unexpected” global values (as defined by the `expectedGlobals` config) and DOM cleanup (the <body>’s innerHTML is cleared). In the Ext JS layer, instead of clearing the DOM, all Ext components created by the test are destroyed.
Results
Of course we’ve tried this new option on the test suites of our own products. For large groups of raw JS tests – ones that don’t involve DOM interaction or event simulation, we’ve seen a major speedup of up to 20 times. Tests that were taking minutes now complete in seconds. For UI tests with user interactions, the speed gain is less obvious due to the internal waiting inside of the tests, but it is still noticeable and such tests are approximately 3-5x times faster.
We’ll be investigating additional possibilities of performance improvements, like running several “chunks” in parallel, but so far the results are very promising. Below are two screenshots of running a group of 68 tests in our suite with sandbox on and off. You can see for yourself why we are so excited about this:
Sandbox enabled (old behavior)
Sandbox disabled
For this group of tests, there is a 15x speed increase compared to before.
Should I use it too?
The disabled sandbox option works very well in our test suites, but there can still be edge cases to polish. We definitely advise you to play with it. Just disable this option for some groups of tests and see if the tests still pass unmodified. Possibly you will need to make some minor changes in your tests. Anyway, please let us know about your experience in our forum!