Get help with testing, discuss unit testing strategies etc.


Post by nilsdehl »

In a current customer project we use Siesta to test an Ext JS modern 6.5.3 application.
All test run green locally in chrome using the siesta UI.
Unit and UI test run smooth in FF and remote on teamcity via slimerjs as well.
But the application tests fail remote in slimerjs while run green if I run them locally in chrome or FF.

I already tried:
runCore: 'sequential',
pauseBetweenTests: 200,
in harness config for the application tests. As result the run more stable in local FF.

Failure example:
030_device_registration.t.js
If wrong server password credentials are used an error message should appear and the user stays on the same view.
fail 5 - Waited too long for: componentQuery: errormessagebox to return a visible component
Failed assertion `waitForComponentQueryVisible`
Condition was not fullfilled during 30000ms
After a succesfull device registration the user should be redirect to the personalisation view.
fail 3 - Waited too long for: componentQuery: personalization to return a visible component
Failed assertion `waitForComponentQueryVisible`
Condition was not fullfilled during 30000ms
Harness:
var harness = new Siesta.Harness.Browser.ExtJS();

harness.configure({
	title: 'Test Suite',
	testClass: CustomTestClass,
..
});

harness.start(
	{
		group: 'Test Suite',
		items: [
			{
				group: 'Unit Tests', // runs stable
				...
			},
                        {
				group: 'UI Tests', // runs stable
				...
			},
			{
				group: 'Application Test',
				pageUrl: '../index.html',
				autoCheckGlobals: false,
				runCore: 'sequential',
				pauseBetweenTests: 200,
				items: [
					'application/010_app_sanity.t.js',
					{
						group: 'Device registration',
						pageUrl: '../#setup/device/registration',
						items: [
							'application/030_device_registration.t.js'
						]
					},
					....
				]
			}
		]
	}
);
030_device_registration.t.js
describe('Device registration', function (t)
{
       // Init sim manager to fake backend 
	t.initFakeBackend();

	t.beforeEach(function (t, _next)
	{
		// clear localstorage to have a clean environment
		localStorage.clear();

		// reset route to registration
		t.setHash('setup/device/registration', _next, t);
	});

	t.it("Device shouldn't be registered", function (t)
	{
		t.expect(dilocsync.core.Master.isRegistered()).toBeFalsy();
	});

	t.it("Device registration view should be rendered", function (t)
	{
		t.chain(
			{
				waitForCQVisible: 'registration'
			}
		);
	});

	t.it("If wrong server password credentials are used an error message should appear and the user stays on the same view.", function (t)
	{
		t.chain(
			{
				waitForCQVisible: 'registration',
				desc: "Registration form should be visible"
			},
			{
				target: 'textfield[name=server]',
				setValue: 'server-does-not-exist'
			},
			{
				target: 'textfield[name=password]',
				setValue: 'wrong-password'
			},
			{
				waitFor: 100,
				desc: "Wait that bind can update the button"
			},
			{
				click: 'button[reference=register] => button'
			},
			{
				waitForCQVisible: 'errormessagebox',
				desc: "Error message should be shown"
			},
			{
				click: 'errormessagebox button => button'
			}
		);
	});
...

});

Most application tests fail at some point on slimerjs on teamcity cause if an "waitForComponentQueryVisible" time out. How can I make the application tests more stable?

Aiming for all tests green everywhere :)

Thanks
Nils

Post by nickolay »

Aiming for all tests green everywhere :)
Great attitude! :)

Try using webdriver + headless firefox/chrome instead of slimerjs? Or puppeteer launcher? There's plenty of headless options now, slimerjs is somewhat outdated project.

Post by nilsdehl »

Thanks Nikolay we will try it with one of the alternatives.

Post Reply