I am running an evaluation of Siesta Trial, version 5.2.2 and I am facing several issues with the code coverage.
I would appreciate help to understand and get it working.
First, I am executing this on 2 environments: Ubuntu 18 (running in VirtualBox) and Windows 10 system.
First of all, I am starting a webserver in the trial folder:
php -S localhost:80
The tests seems to be running, but I am trying to understand how the code coverage works, but that seems problematic.
I execute the following code and get different results on linux/windows:
(Btw, it runs much faster if --nyc.include is fine-tuned to exclude the "ext-all" from instrumentation, like `**/5.code_coverage/**/*.js`)
In your case it seems only the "ext-all" is included in the report, which makes me think this something with the internal proxy, that Siesta uses for instrumentation.
I'm running local Apache instance and thats probably a bit different from using php built-in server.
Can you please run this command and post the output?
[DEBUG] Dispatcher start
[DEBUG] Starting instrumentation proxy
[DEBUG] HTTP Instrumentation proxy has started on :::42891
[DEBUG] HTTPS Instrumentation proxy has started on :::40349
[DEBUG] Runner setup: a Siesta.Launcher.Runner.WebDriverNodeJS
[DEBUG] Dispatcher setup starting
[DEBUG] Trying to create a WebDriver instance for browser: chrome
[DEBUG] WebDriver instantiated successfully
[DEBUG] PageId: 1, method : getConfigInfo starting, args size: 30
[DEBUG] PageId: 1, method : getConfigInfo, took 0.021s, result size : 2347
[DEBUG] Dispatcher setup completed, launching the suite
Launching test suite, OS: Linux, agent: Chrome 74.0.3729.169
[DEBUG] Launch runner, pagesCount: 1, max : 1, pages left: 1
[DEBUG] PageId: 1, method : launchAutomatedTests starting, args size: 428
[DEBUG] PageId: 1, method : launchAutomatedTests, took 0.057s, result size : 4
[DEBUG] Page polling has started
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/resources/css/ext-all.css will instrument: false
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/ext-all-debug.js will instrument: false
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/resources/ext-theme-classic/ext-theme-classic-all.css will instrument: false
[DEBUG] PageId: 1, method : getAutomationState starting, args size: 2
[DEBUG] PageId: 1, method : getAutomationState, took 0.306s, result size : 2626
[PASS] 5.code_coverage/010_range.t.js
[PASS] 5.code_coverage/020_event.t.js
[DEBUG] PageId: 1, method : getAutomationState starting, args size: 2
[DEBUG] PageId: 1, method : getAutomationState, took 0.02s, result size : 2362
[PASS] 5.code_coverage/030_loader.t.js
[PASS] 5.code_coverage/040_pageUrl.t.js
[DEBUG] Recevied results for all tests in the suite, proceeding to finalization
7 passed, 0 failed assertions took 3.885s to complete
[DEBUG] Generating reports (if any)
----------|----------|----------|----------|----------|-------------------|
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s |
----------|----------|----------|----------|----------|-------------------|
All files | 0 | 0 | 0 | 0 | |
----------|----------|----------|----------|----------|-------------------|
[DEBUG] Reports processed (if any)
[DEBUG] Page close started: 1
I ran this with Apache, I thought maybe it makes a difference, but the result is the same. Same result on windows.
Let me know if you need more info. I'll wait for your feedback.
Re: Siesta Trial - code coverage issue
Posted: Mon May 27, 2019 1:19 pm
by nickolay
Just a wild guess, if you are blocked by this (I'm still fixing the firefox automation in trial) - perhaps the problem is, that Chrome has a setting "bypass proxy for localhost" enabled? In my setup I have different host name 'lh', aliased in /etc/hosts, thats why it works for me and requests goes through the proxy.
You can try to alias the host using different host name, or wait some time.
Re: Siesta Trial - code coverage issue
Posted: Mon May 27, 2019 1:23 pm
by klodoma
I am not blocked by this. As I wrote I am running an evaluation.
My current target is to understand how the code coverage works and for that I would need to see the results. But it's not blocking anything unless you tell me we'll find a solution in 2 months
I'll give it a shot with another host name and let you know.
Re: Siesta Trial - code coverage issue
Posted: Mon May 27, 2019 2:00 pm
by klodoma
Ok, the localhost proxy did something.
I don't seem to get the --nyc.include working properly, it's not matching anything...
With --nyc.all I get some coverage report (haven't checked it yet in detail)
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/010_range.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/020_event.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/030_loader.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/040_pageUrl.t.js will instrument: false
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/resources/css/ext-all.css will instrument: false
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/ext-all-debug.js will instrument: false
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/resources/ext-theme-classic/ext-theme-classic-all.css will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/several_classes_in_one_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/some_folder/preload_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/some_folder/folder2/preload_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/010_range.t.js will instrument: false
[DEBUG] PROXYING request for https://local/resources/css/font-awesome/fonts/fontawesome-webfont.woff2?v=4.4.0 will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/several_classes_in_one_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/some_folder/preload_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/some_folder/folder2/preload_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/020_event.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/several_classes_in_one_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/030_loader.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/lib/My/Store/EventStore.js?_dc=1558961938247 will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/lib/My/Store/Base.js?_dc=1558961938261 will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/lib/My/Util/SomeClass.js?_dc=1558961938278 will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/pageUrl.html will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/040_pageUrl.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/lib/My/Store/ResourceStore.js?_dc=1558961938429 will instrument: false
it seems that the filtering --nyc.include='**/*code_coverage*/*.js' is not forwared properly or smth like that