Hi,
I want to immediately exit a test when any failure in any spawned subtest (describe, it, chain, ...) occurs. I know about the breakOnFail config but this will exit the whole project (i.e. when having multiple test classes the next one is not run).
Is there any config I missed or another option to enforce this behaviour?
Support Forum
Hi,
Please check if https://www.bryntum.com/docs/siesta/#!/api/Siesta.Test-method-exit suites your need?
Please check if https://www.bryntum.com/docs/siesta/#!/api/Siesta.Test-method-exit suites your need?
Read the API documentation
I don't think we have "breakOnFail" for individual tests, but probably it can be implemented manually, by listening to test update events.
Read the API documentation
So sadly no missed config. Ok, time to dig deeper. I played around with the exit function but always found a constellation that did not meet my requirements. For example this test:
I expected the test to exit and be done but unfortunately the exit method throws an unhandled __SIESTA_TEST_EXIT_EXCEPTION__ error. Wrapping the exit call in try/catch will produce the desired output but I do not see the point of a framework function that you manually have to try/catch.
The next example is this:
Again, expected behaviour would be that the test simply stops when calling exit. This time however 'Subtest 3' is started before exiting (without test 6 being evaluated) and a javascript error is thrown: 'Uncaught TypeError: data.next is not a function'. And of course the warning 'Adding assertions after the test has finished' is shown. Note that 'Subtest 3' is not started if you remove the first waitFor in the chain.
Changing 'Subtest 2' gives yet another behaviour:
Now the javascript error 'Uncaught TypeError: data.next is not a function' is not thrown but all the other problems mentioned above still occur.
me.test.describe('Test with multiple steps', function(st) {
st.it('Subtest 1', function(sst) {
sst.is(1, 1);
});
st.it('Subtest 2', function(sst) {
sst.is(2, 'mustFail');
me.test.exit('failed');
sst.is(3, 3);
});
st.it('Subtest 3', function(sst) {
sst.is(4, 4);
sst.is(5, 5);
});
});
The next example is this:
me.test.describe('Test with multiple steps', function(st) {
st.it('Subtest 1', function(sst) {
sst.is(1, 1);
});
st.it('Subtest 2', function(sst) {
sst.chain({
waitFor: 10
}, function(next) {
sst.is(2, 'mustFail');
try { me.test.exit('failed'); } catch (e) {}
sst.is(3, 3);
next();
}, {
waitFor: 10
});
});
st.it('Subtest 3', function(sst) {
sst.chain(function(next) {
sst.is(4, 4);
sst.is(5, 5);
next();
}, {
waitFor: 10
}, function() {
sst.is(6, 6);
});
});
});
Changing 'Subtest 2' gives yet another behaviour:
st.it('Subtest 2', function(sst) {
sst.chain({
waitFor: 10
}, function(next) {
sst.waitForElementVisible('foo', next);
sst.is(2, 'mustFail');
try { me.test.exit('failed'); } catch (e) {}
sst.is(3, 3);
}, {
waitFor: 10
});
});
Well, the only way to stop the execution of the function is to throw an exception (or `return` a value). The exception that is thrown is handled in a special way, that it does not even appear in the console. For example, for this test:
I have this result:
And for this test, with exit using top-level `t` instance, the 3rd `it` is not launched:
No exception in console in both cases.
It seems your setup is different from "plain" Siesta, can you describe how exactly and why do you observe the exception for test exit?
StartTest(function (t) {
t.describe('Test with multiple steps', function (st) {
st.it('Subtest 1', function (sst) {
sst.is(1, 1);
});
st.it('Subtest 2', function (sst) {
sst.is(2, 'mustFail');
sst.exit('failed');
sst.is(3, 3);
});
st.it('Subtest 3', function (sst) {
sst.is(4, 4);
sst.is(5, 5);
});
});
})
StartTest(function (t) {
t.describe('Test with multiple steps', function (st) {
st.it('Subtest 1', function (sst) {
sst.is(1, 1);
});
st.it('Subtest 2', function (sst) {
sst.is(2, 'mustFail');
t.exit('failed');
sst.is(3, 3);
});
st.it('Subtest 3', function (sst) {
sst.is(4, 4);
sst.is(5, 5);
});
});
})
It seems your setup is different from "plain" Siesta, can you describe how exactly and why do you observe the exception for test exit?
Read the API documentation
Hm.. Just did, looks the same for me? (using the `siesta/examples/browser` suite)
Read the API documentation
It makes the difference, yes. However, this is the intention of the "transparent exceptions" mode - to not catch any exceptions, to ease the debugging (otherwise one would have to use "stop on caught exceptions" debugger mode). These are a bit contradicting requirements, I think the ease of debugging is important, so this behavior can't be changed easily.
Read the API documentation