2

Closed

Form submission causes test run to hang and eventually timeout

description

We have a test suite which has recently grown to just over 200 tests. Recently I have started experiencing an issue on our build server whereby the test run appears to hang at around 150 tests and subsequently bombs out with a ChutzpahTimeoutException. Interestingly the number of tests after which it hangs varies between runs, but is always around 150.

We can run the tests locally fine, but our development machines are quite powerful compared to the build server.

I am using Chutzpah 2.2.1.173 and our tests are written in Jasmine using RequireJS.

file attachments

Closed Mar 10 at 8:48 PM by mmanela

comments

liddellj wrote Jan 18, 2013 at 2:07 PM

This definitely appears to be system resource related.

I tried splitting our test suite up into two separate runs. The first half worked but the second exhibited the same issue. I attempted to narrow it down to a specific test but couldn't - it definitely seems to be related to the number of tests. I then tried splitting the test suite into 3, and our build is now * occasionally* working. I suspect this may be due to the physical machine on which the build slave runs being under less load this afternoon (it's a VM).

So my current gut feel is that Chutzpah does not behave well on a slow system (the CPU is maxed out). I have also tried upping the timeoutMilliseconds parameter - but this does not appear to change the point at which the test run hangs, but only the time after the hang before the timeout exception comes back.

mmanela wrote Jan 19, 2013 at 10:46 PM

That is interesting. I know of teams that are running Chutzpah on suites of several thousand unit tests. Are you running the tests in parallel? On the machine the tests are running on do you see both memory and CPU spiking?

If you look at the process monitor do you see instanced of chutzpah or phantom hanging around?

I would love to try to help of figure this out, would it be possible to share with me your suite of units tests? Or at least a set of unit tests that reproduce the issue?

liddellj wrote Jan 21, 2013 at 11:31 AM

I've attached out test suite. The command I'm running is:
 chutzpah.console.exe ./test-main.js
I'm not running in parallel (although I did try it - same issue) - the machine only has one core anyway. The test run freezes at around 150 tests (varies between runs). There are no processed left around after the run, both chutzpah and phantomjs appear to exit correctly. During the run, CPU appears to max out, but not memory. I notice that during the run, phantomjs.exe appears to have a very high CPU usage for the duration of the run (even after the apparent 'hang' on the command line.)

If there's anything else you need to know please let me know.

liddellj wrote Jan 24, 2013 at 3:16 PM

Have you had any chance to look at the test suite yet? If I get the time I may try to re-create the issue in a VM.

mmanela wrote Jan 24, 2013 at 3:52 PM

I haven't had a chance to look at it. I will update once I do.

liddellj wrote Jan 30, 2013 at 7:55 PM

One of my colleagues took a look at this and was able to narrow it down to a test which submitted a form ($('.selector').submit();). The submit event was subsequently handled by a function defined within a test class but we were not calling e.preventDefault();. Adding this statement stopped the tests from hanging.

So my guess is that the form submission causes a HTTP request within PhantomJS during the test run, which fails as there is nothing there to respond to it - and at that point for whatever reason it causes the entire test run to hang and timeoue. This would explain why the point at which it fails varied - presumably it depends how long it take the HTTP request to fail.

Obviously we shouldn't be doing a HTTP request from our unit tests, but I'm still surprised that this causes the entire run to freeze. Not sure if this is an issue with Chutzpah, Jasmine or PhantomJS?

Should be fairly easy to recreate.

coatesjable wrote Jan 31, 2013 at 7:56 AM

One of my colleagues took a look at this and was able to narrow it down to a test which submitted a form ($('.selector').submit();).
I was that co-worker.

We are using requirejs to load all of our spec files. The entry point for the test runner is a single file, let's say test-main.js. The code below shows that you like for actual files specified when you launch the tool:
   Parallel.ForEach(fileProbe.FindScriptFiles(testPaths, options.TestingMode), parallelOptions, testFile =>
            {
                try
                {
...
                    if (testContextBuilder.TryBuildContext(testFile, out testContext))
                    {
                        var testSummary = InvokeTestRunner(headlessBrowserPath,
                                                           options,
                                                           testContext,
                                                           testRunnerMode,
                                                           callback);
                        testFileSummaries.Enqueue(testSummary);

                }
                catch (Exception e)
                {
                    callback.ExceptionThrown(e, testFile.FullPath);
                }
            });
As we use requirejs for spec loading the find files method will only find a singe file (of course) which would then be passed to phantomjs that will then pull in the rest of the specs. I think that when our test times out phantomjs stalls.

I think Chutzpah is working as designed and behaves sensibly when phantomjs times out.

What might be nice though would be for chutzpah to load the spec files using requirejs before passing them to phantomjs. That way we don't lose the entire test run if phantomjs times out. The FindScriptFiles method might be able to return a list of files that were loaded via requirejs. That way you could iterate over the set (possible in parallel).

I know that you have this functionality now (minus requirejs) if we point the tool at a directory but I think it would be great to consider test suites that use AMD loading.

You could say that phantomjs should not abandon the test run for a single timeout but hey that's what we have right now.

Hopefully I haven't misrepresented the tool. We think it's great and are keen to help.

mmanela wrote Feb 7, 2013 at 5:01 PM

You make a good point. If you are loading are files with AMD then Chutzpah looks at is as one big test run and the timeout will kill it all. It would be nice to run them individually but to do so would require parsing require() paths. I haven;t thought much about this and am not sure how well it would work.

mmanela wrote Dec 3, 2013 at 4:44 AM

In the upcoming release Chutzpah will allow you to run amd test individually and it should alleviate this issue.

mmanela wrote Dec 13, 2013 at 7:23 PM

The improved AMD support has been released in version 3.0

mmanela wrote Dec 14, 2013 at 8:57 PM

Using new functionality in Chutzpah 3.0 I was able to get your repro to work! I attached an updated version of your sample (fixed.zip).

The key addition is adding the chutzpah.json file with the following contents:
{
    "TestHarnessReferenceMode": "AMD",
    "TestHarnessLocationMode": "SettingsFileAdjacent",
    "References" : [
        {"Path" : "require.js"},
        {"Path" : "test-main.js"}
    ]
}
With this in place you can run tests like:
chutzpah.console.exe tests\collections\SearchResultsSpecs.js