Get help with testing, discuss unit testing strategies etc.


Post by paulb »

Hi,
I want to immediately exit a test when any failure in any spawned subtest (describe, it, chain, ...) occurs. I know about the breakOnFail config but this will exit the whole project (i.e. when having multiple test classes the next one is not run).
Is there any config I missed or another option to enforce this behaviour?


Post by nickolay »


Post by nickolay »

I don't think we have "breakOnFail" for individual tests, but probably it can be implemented manually, by listening to test update events.

Post by paulb »

So sadly no missed config. Ok, time to dig deeper. I played around with the exit function but always found a constellation that did not meet my requirements. For example this test:
me.test.describe('Test with multiple steps', function(st) {
    st.it('Subtest 1', function(sst) {
        sst.is(1, 1);
    });

    st.it('Subtest 2', function(sst) {
        sst.is(2, 'mustFail');
        me.test.exit('failed');
        sst.is(3, 3);
    });

    st.it('Subtest 3', function(sst) {
        sst.is(4, 4);
        sst.is(5, 5);
    });
});
I expected the test to exit and be done but unfortunately the exit method throws an unhandled __SIESTA_TEST_EXIT_EXCEPTION__ error. Wrapping the exit call in try/catch will produce the desired output but I do not see the point of a framework function that you manually have to try/catch.

The next example is this:
me.test.describe('Test with multiple steps', function(st) {
    st.it('Subtest 1', function(sst) {
        sst.is(1, 1);
    });

    st.it('Subtest 2', function(sst) {
        sst.chain({
            waitFor: 10
        }, function(next) {
            sst.is(2, 'mustFail');
            try { me.test.exit('failed'); } catch (e) {}
            sst.is(3, 3);
            next();
        }, {
            waitFor: 10
        });
    });

    st.it('Subtest 3', function(sst) {
        sst.chain(function(next) {
            sst.is(4, 4);
            sst.is(5, 5);
            next();
        }, {
            waitFor: 10
        }, function() {
            sst.is(6, 6);
        });
    });
});
Again, expected behaviour would be that the test simply stops when calling exit. This time however 'Subtest 3' is started before exiting (without test 6 being evaluated) and a javascript error is thrown: 'Uncaught TypeError: data.next is not a function'. And of course the warning 'Adding assertions after the test has finished' is shown. Note that 'Subtest 3' is not started if you remove the first waitFor in the chain.

Changing 'Subtest 2' gives yet another behaviour:
    st.it('Subtest 2', function(sst) {
        sst.chain({
            waitFor: 10
        }, function(next) {
            sst.waitForElementVisible('foo', next);
            sst.is(2, 'mustFail');
            try { me.test.exit('failed'); } catch (e) {}
            sst.is(3, 3);
        }, {
            waitFor: 10
        });
    });
Now the javascript error 'Uncaught TypeError: data.next is not a function' is not thrown but all the other problems mentioned above still occur.

Post by nickolay »

Well, the only way to stop the execution of the function is to throw an exception (or `return` a value). The exception that is thrown is handled in a special way, that it does not even appear in the console. For example, for this test:
StartTest(function (t) {
    t.describe('Test with multiple steps', function (st) {
        st.it('Subtest 1', function (sst) {
            sst.is(1, 1);
        });

        st.it('Subtest 2', function (sst) {
            sst.is(2, 'mustFail');
            sst.exit('failed');
            sst.is(3, 3);
        });

        st.it('Subtest 3', function (sst) {
            sst.is(4, 4);
            sst.is(5, 5);
        });
    });
})
I have this result:
Siesta generic browser examples - Google Chrome_007.png
Siesta generic browser examples - Google Chrome_007.png (319.07 KiB) Viewed 3728 times
And for this test, with exit using top-level `t` instance, the 3rd `it` is not launched:
StartTest(function (t) {

    t.describe('Test with multiple steps', function (st) {
        st.it('Subtest 1', function (sst) {
            sst.is(1, 1);
        });

        st.it('Subtest 2', function (sst) {
            sst.is(2, 'mustFail');
            t.exit('failed');
            sst.is(3, 3);
        });

        st.it('Subtest 3', function (sst) {
            sst.is(4, 4);
            sst.is(5, 5);
        });
    });
})
Siesta generic browser examples - Google Chrome_008.png
Siesta generic browser examples - Google Chrome_008.png (317.4 KiB) Viewed 3728 times
No exception in console in both cases.

It seems your setup is different from "plain" Siesta, can you describe how exactly and why do you observe the exception for test exit?

Post by paulb »

The 'Subtest 3' is only launched for my second example using chain and waitFor. Could you elaborate on that?

I will look into why the exception is thrown in my case.

Post by paulb »

Found out why the exception is thrown. You will not believe me until you tried it yourself. Try removing the title config in the configure call of the project...

Post by nickolay »

Hm.. Just did, looks the same for me? (using the `siesta/examples/browser` suite)

Post by paulb »

Did you check the transparentExceptions options in the GUI?

Edit: Oh well, looks like adding/removing the title will reset the checked options in the GUI. That makes sense. So the thrown exception mystery is solved for me.

Post by nickolay »

It makes the difference, yes. However, this is the intention of the "transparent exceptions" mode - to not catch any exceptions, to ease the debugging (otherwise one would have to use "stop on caught exceptions" debugger mode). These are a bit contradicting requirements, I think the ease of debugging is important, so this behavior can't be changed easily.

Post Reply