Allow failure reports after command line run


In a project that I'm working on we currently have around 300 very verbose tests. Chutzpah is integrated into the build system so we run the following command at build time:
chutzpah.console.exe myTestSuite.html /silent /debug /timeoutMilliseconds 30000
Now the output of the tests is very verbose so the console is then flooded with log messages (a good thing). On occasion there'll be failing tests and the console will contain something like "x number of tests failed" etc. at the very end of the exe run. This is great in determining that tests failed but not so great in determining which tests and why they failed.

In essence I could scroll up and look through the several thousand lines of debug output in an attempt to find out what tests failed and why but with so many lines of output it becomes pretty impossible.

It'd be great if the debug option also re-output the failing tests and their corresponding logs at the end of the test run. This way all failures are in one location and are easy to decipher potential failure causes.
Closed Dec 3, 2013 at 4:39 AM by mmanela


damianedwards wrote Jul 30, 2013 at 4:32 AM

This would really help us on the SignalR project too. This is a common pain we endure.

mmanela wrote Jul 30, 2013 at 3:13 PM

First of all, the debug flag is not intended for this purpose. I understand why you are using it but it is more for diagnostic purposes. For example, if a user is having issues it will show the raw info of what Chutzpah is getting. In the next release it will get noisier so it will not be a great flag to use.

That said I would like to make it so the non-debug output is more helpful. So I can either make it default that I print an error report at the end. Does that sound good? Or should I put this behind a flag like /errorreport?

Is there anything else from the debug output you are using?

NTaylorMullen wrote Jul 31, 2013 at 5:06 AM

I definitely think defaulting to the error report would be best. There's nothing else we're really using from the debug output, it was just the only way to get the tests console.logged information.

mmanela wrote Jul 31, 2013 at 2:46 PM

Yea, I already checked in (or will soon) a change which outputs the logging information by default also

mmanela wrote Oct 3, 2013 at 3:21 AM

I checked in a new feature to address this.

This adds a new command line flag /showFailureReport that when given will show a summary of the failures.

NTaylorMullen wrote Oct 3, 2013 at 4:08 AM

Awesome! We'll have to give it a try :)

mmanela wrote Oct 3, 2013 at 5:10 AM

This will be included in the next release which I hope to do in the next couple weeks. I am waiting on a PR to come in and test out.

mmanela wrote Dec 13, 2013 at 7:18 PM

This feature has now been release in version 3.0