This definitely appears to be system resource related.

I tried splitting our test suite up into two separate runs. The first half worked but the second exhibited the same issue. I attempted to narrow it down to a specific test but couldn't - it definitely seems to be related to the number of tests. I then tried splitting the test suite into 3, and our build is now * occasionally* working. I suspect this may be due to the physical machine on which the build slave runs being under less load this afternoon (it's a VM).

So my current gut feel is that Chutzpah does not behave well on a slow system (the CPU is maxed out). I have also tried upping the timeoutMilliseconds parameter - but this does not appear to change the point at which the test run hangs, but only the time after the hang before the timeout exception comes back.

That is interesting. I know of teams that are running Chutzpah on suites of several thousand unit tests. Are you running the tests in parallel? On the machine the tests are running on do you see both memory and CPU spiking?

If you look at the process monitor do you see instanced of chutzpah or phantom hanging around?

I would love to try to help of figure this out, would it be possible to share with me your suite of units tests? Or at least a set of unit tests that reproduce the issue?

I've attached out test suite. The command I'm running is:

 chutzpah.console.exe ./test-main.js

I'm not running in parallel (although I did try it - same issue) - the machine only has one core anyway. The test run freezes at around 150 tests (varies between runs). There are no processed left around after the run, both chutzpah and phantomjs appear to exit correctly. During the run, CPU appears to max out, but not memory. I notice that during the run, phantomjs.exe appears to have a very high CPU usage for the duration of the run (even after the apparent 'hang' on the command line.)

If there's anything else you need to know please let me know.

Have you had any chance to look at the test suite yet? If I get the time I may try to re-create the issue in a VM.

I haven't had a chance to look at it. I will update once I do.

One of my colleagues took a look at this and was able to narrow it down to a test which submitted a form ($('.selector').submit();). The submit event was subsequently handled by a function defined within a test class but we were not calling e.preventDefault();. Adding this statement stopped the tests from hanging.

So my guess is that the form submission causes a HTTP request within PhantomJS during the test run, which fails as there is nothing there to respond to it - and at that point for whatever reason it causes the entire test run to hang and timeoue. This would explain why the point at which it fails varied - presumably it depends how long it take the HTTP request to fail.

Obviously we shouldn't be doing a HTTP request from our unit tests, but I'm still surprised that this causes the entire run to freeze. Not sure if this is an issue with Chutzpah, Jasmine or PhantomJS?

Should be fairly easy to recreate.

One of my colleagues took a look at this and was able to narrow it down to a test which submitted a form ($('.selector').submit();).

I was that co-worker.

We are using requirejs to load all of our spec files. The entry point for the test runner is a single file, let's say test-main.js. The code below shows that you like for actual files specified when you launch the tool:

   Parallel.ForEach(fileProbe.FindScriptFiles(testPaths, options.TestingMode), parallelOptions, testFile =>
            {
                try
                {
...
                    if (testContextBuilder.TryBuildContext(testFile, out testContext))
                    {
                        var testSummary = InvokeTestRunner(headlessBrowserPath,
                                                           options,
                                                           testContext,
                                                           testRunnerMode,
                                                           callback);
                        testFileSummaries.Enqueue(testSummary);

                }
                catch (Exception e)
                {
                    callback.ExceptionThrown(e, testFile.FullPath);
                }
            });

As we use requirejs for spec loading the find files method will only find a singe file (of course) which would then be passed to phantomjs that will then pull in the rest of the specs. I think that when our test times out phantomjs stalls.

I think Chutzpah is working as designed and behaves sensibly when phantomjs times out.

What might be nice though would be for chutzpah to load the spec files using requirejs before passing them to phantomjs. That way we don't lose the entire test run if phantomjs times out. The FindScriptFiles method might be able to return a list of files that were loaded via requirejs. That way you could iterate over the set (possible in parallel).

I know that you have this functionality now (minus requirejs) if we point the tool at a directory but I think it would be great to consider test suites that use AMD loading.

You could say that phantomjs should not abandon the test run for a single timeout but hey that's what we have right now.

Hopefully I haven't misrepresented the tool. We think it's great and are keen to help.

You make a good point. If you are loading are files with AMD then Chutzpah looks at is as one big test run and the timeout will kill it all. It would be nice to run them individually but to do so would require parsing require() paths. I haven;t thought much about this and am not sure how well it would work.