Javascript Unit Tests on Team Foundation Service with Chutzpah

Oct 25, 2012 at 5:37 PM
Edited Oct 25, 2012 at 6:46 PM

There is a great post about running TFS Builds with Chutzpah at http://blogs.msdn.com/b/visualstudioalm/archive/2012/07/09/javascript-unit-tests-on-team-foundation-service-with-chutzpah.aspx.  But I am running into an issue with it not finding any tests.

No test is available in C:\Builds\4\..\Scripts\Test\site\tests.  Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

I checked on the server, and the files are there, any ideas?

Oct 25, 2012 at 6:42 PM

Here is the real error that is coming back.

Error: Method 'ToXml' in type 'Chutzpah.VS2012.TestAdapter.ChutzpahAdapterSettings' from assembly 'Chutzpah.VS2012.TestAdapter, Version=2.1.0.162, Culture=neutral, PublicKeyToken=1ca802c37ffe1896' does not have an implementation.

Coordinator
Oct 25, 2012 at 7:18 PM

That is a really strange error message. By itself it doesn't make much sense to me. Is there any stack trace or other information with that error?

Can you setup a simple solution that will let me reproduce the issue?

Oct 25, 2012 at 7:38 PM

Unfortunately, there is no stack trace from the Build Server in this case.  I have run the Chutzpah extension locally on the same solution and it works fine.

Oct 25, 2012 at 8:06 PM

I found that the ToXml error that I was getting was only coming back when I tried to edit the build definition.  Here is the whole message when I try to edit the definition.

 

Summary: There were 0 failures, 1 errors and 1 warnings loading custom activities and services.

Error: Method 'ToXml' in type 'Chutzpah.VS2012.TestAdapter.ChutzpahAdapterSettings' from assembly 'Chutzpah.VS2012.TestAdapter, Version=2.1.0.162, Culture=neutral, PublicKeyToken=1ca802c37ffe1896' does not have an implementation.

Warning: Could not load file or assembly 'file:///C:\Users\...\AppData\Local\Temp\VSTFSBuild\...\...\phantomjs.exe' or one of its dependencies. The module was expected to contain an assembly manifest.

Oct 26, 2012 at 2:10 PM
Edited Oct 26, 2012 at 2:10 PM

Here is a link to my SkyDrive with a solution that contains a simple test file to reproduce the issue.

http://sdrv.ms/UMbUN7

Nov 1, 2012 at 2:23 PM

We are having very similar issues. After following the same blog post mentioned above, we get the same two errors (Method 'ToXml' and 'phantonjs.exe or one of it's dependencies...'). We've configured two test runs in the build definition - one for test*.dll and second one for **\*.js - and from a practical standpoint, it seems the test runner simply does not execute the second run, only the first.

We're kind of stuck at this point...

Coordinator
Nov 1, 2012 at 2:27 PM

That is odd. I have never run it in TFS Build before so I need to test this out and see what is going on.  

Once I get some time I will dig into this further.

Nov 1, 2012 at 2:30 PM

That would great.  JavaScript is a bit part of the projects we do now, and with continuous delivery, we want to make sure tests can be run automatically on our build server.  Thanks again.

Coordinator
Nov 5, 2012 at 5:12 PM

I am unable to reproduce your issue. I followed the steps outlines in http://blogs.msdn.com/b/visualstudioalm/archive/2012/07/09/javascript-unit-tests-on-team-foundation-service-with-chutzpah.aspx?CommentPosted=true#commentmessage and I was able to run the tests successfully as part of a integrated build.

 

Here are the steps I followed:

1. Unziped the solution you attached

2. Downloaded the Chutzpah Test Adapter from http://visualstudiogallery.msdn.microsoft.com/f8741f04-bae4-4900-81c7-7c9bfb9ed1fe

3. I unziped the test adapter contents into a folder named Chutzpah which I included in the solution.

4. I followed the steps in the blog post to set the Chutzpah folder as the "Version Control Path to custom Assemblies"

5. I updated the test assembly file specification to **\tests.js

6. I *did* not set the tests.js to "Copy Always" as the blog said. When I did this it ran that file multiple times for some reason.

7. After these steps I was able to checkin and run the build successfully.

 

 

Did you procedure differ?

Nov 5, 2012 at 5:14 PM
Edited Nov 5, 2012 at 5:15 PM

Did you copy all files from Chutzpah to your build assemblies folder, or only the ones outlined in the blog post?

Nov 5, 2012 at 5:16 PM

When you ran the build, did it show you the JavaScript tests in the test results?  The build ran successfully for me to, but it would not show any results for the tests.  When I open the build definition it shows the errors described above.  Thanks for looking into this.

Coordinator
Nov 5, 2012 at 5:25 PM

1. I copied *all* the files from the .vsix.

2. It did report the test results. I then updated one of the tests to have a failing assert and it detected that correctly also.

Nov 5, 2012 at 5:28 PM

It seems that rexmorgan and I have the same problem.  I know that I put in two test runs as well, one for Test.*dll and one for **\*.js.  Does that fail your results if you add the additional test run?

Nov 5, 2012 at 5:31 PM

I started over with a fresh solution & build definition, repeated the steps above* and still seeing the same behavior - the test run which specifies .js simply does not fire. If we have two test runs (one for .dll, one for .js), we get "1 test run completed" - with the .NET unit test results shown. If the .dll test run is removed, we get "No Test Results".

*Which is more or less what we had previously (mainly just keeping the Chutzpah files in a central location instead of inside the actual test project).

Nov 5, 2012 at 5:33 PM

@rexmorgan

The solution I posted at http://sdrv.ms/UMbUN7 only contains JavaScript unit tests.  Does that solution work for you?

Coordinator
Nov 5, 2012 at 5:39 PM

The setup I did has only 1 test run and it was looking just for **\tests.js. I can try later with adding multiple runs. 

Nov 5, 2012 at 5:42 PM

@mmanela

My test runs specify **\*.js.  It could be that when it does not receive a specific file to test it fails.  Or maybe if it picks up any files that are not unit tests, it fails.

Nov 5, 2012 at 6:02 PM
Edited Nov 5, 2012 at 7:02 PM

I repeated the steps precisely and also introduced the additional conditions that makes our setup different and they both fail in exactly the same way (js tests simply do not run) - do there must be an environmental issue that's different. Are you running .NET 4.5 on the build server?

Coordinator
Nov 5, 2012 at 6:17 PM

I ran my build on the hosted tfs (tfs.visualstudio.com). I am not sure if the build agents there are different.

Nov 5, 2012 at 6:24 PM

It seems like it could be related to an assembly difference in Microsoft.VisualStudio.TestPlatform.ObjectModel.dll.

Nov 5, 2012 at 6:25 PM

The problem is at VS2012.TestAdapter\ChutzpahAdapterSettings.cs in the ToXml method.  It could be the override does not match that of the loaded assembly of Microsoft.VisualStudio.TestPlatform.ObjectModel.dll.

Coordinator
Nov 5, 2012 at 6:31 PM

That seems plausible. What version of VS/TFS are you using?


Nov 5, 2012 at 6:33 PM

We are using TFS 2012 and VS 2012 with .NET 4.5.

Nov 5, 2012 at 6:34 PM
Edited Nov 5, 2012 at 6:34 PM

When I put Microsoft.VisualStudio.TestPlatform.ObjectModel.dll in the custom assemblies folder, the Build Definition gave the error that that assembly was already loaded, so I removed it from the folder.

Nov 5, 2012 at 6:46 PM
Edited Nov 5, 2012 at 7:02 PM

We removed that dll thinking the same, no difference. Also on 2012 / 4.5

Nov 5, 2012 at 7:25 PM

@mmanela

Do you get an error with the "Microsoft.VisualStudio.TestPlatform.ObjectModel.dll" assembly when you try to edit your build definition?

Coordinator
Nov 5, 2012 at 7:49 PM

No I don't get any error.

Dec 30, 2012 at 3:59 PM

I'm getting the same exact problem.  It looks like I have a newer version of the Microsoft.VisualStudio.TestPlatform.ObjectModel.dll on my machine.  My machine has version 11.0.51106.1 in  C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow.  Whereas the adapter seems to be using version 11.0.50907.0.  It looks like the signature for ToXml hasn't changed from what you've implemented so must be a DLL hell problem?  Maybe if if the app.config allowed a binding redirect?

<runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.TestPlatform.ObjectModel" publicKeyToken="whatever" culture="neutral" />
        <bindingRedirect oldVersion="11.0.50907.0" newVersion="11.0.51106.1" />
      </dependentAssembly>

Mar 4, 2013 at 2:25 PM
I moved this to an Issue, so that it could be followed up on a little easier.

Issue 98: Method 'ToXml' in type 'Chutzpah.VS2012.TestAdapter.ChutzpahAdapterSettings' does not have an implementation

This is a great addition to be able to run Chutzpah in the Build Server. Hopefully, we can find a way to get it to work. I will see if I can download the source and try building it with the Microsoft.VisualStudio.TestPlatform.ObjectModel.dll assembly from our build server.
Jun 4, 2013 at 5:43 AM
When I tried to edit an build definition, I got following errors in the output window. It seems that Chutzpah is referencing assemblies from VS 2010, not VS 2012.

Summary: There were 0 failures, 4 errors and 1 warnings loading custom activities and services.
Error: Could not load file or assembly 'Microsoft.VisualStudio.Shell.10.0, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
Error: Could not load file or assembly 'file:///C:\Users\xxxxxx\AppData\Local\Temp\VSTFSBuild\3d8d2dd3-7675-4b14-a758-b34826a2c5dc\76770201\Microsoft.VisualStudio.Shell.10.0.dll' or one of its dependencies. The system cannot find the file specified.
Error: Method 'ToXml' in type 'Chutzpah.VS2012.TestAdapter.ChutzpahAdapterSettings' from assembly 'Chutzpah.VS2012.TestAdapter, Version=2.4.1.266, Culture=neutral, PublicKeyToken=1ca802c37ffe1896' does not have an implementation.
Error: API restriction: The assembly 'file:///C:\Users\xxxxxx\AppData\Local\Temp\VSTFSBuild\3d8d2dd3-7675-4b14-a758-b34826a2c5dc\76770201\Microsoft.VisualStudio.TestPlatform.ObjectModel.dll' has already loaded from a different location. It cannot be loaded from a new location within the same appdomain.
Warning: Could not load file or assembly 'file:///C:\Users\xxxxxx\AppData\Local\Temp\VSTFSBuild\3d8d2dd3-7675-4b14-a758-b34826a2c5dc\76770201\phantomjs.exe' or one of its dependencies. The module was expected to contain an assembly manifest.
Summary: There were 0 failures, 4 errors and 1 warnings loading custom activities and services.
Error: Could not load file or assembly 'Microsoft.VisualStudio.Shell.10.0, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
Error: Could not load file or assembly 'file:///C:\Users\xxxxxx\AppData\Local\Temp\VSTFSBuild\cbce9143-7133-45af-a385-0c25ab072cef\76770201\Microsoft.VisualStudio.Shell.10.0.dll' or one of its dependencies. The system cannot find the file specified.
Error: Method 'ToXml' in type 'Chutzpah.VS2012.TestAdapter.ChutzpahAdapterSettings' from assembly 'Chutzpah.VS2012.TestAdapter, Version=2.4.1.266, Culture=neutral, PublicKeyToken=1ca802c37ffe1896' does not have an implementation.
Error: API restriction: The assembly 'file:///C:\Users\xxxxxx\AppData\Local\Temp\VSTFSBuild\cbce9143-7133-45af-a385-0c25ab072cef\76770201\Microsoft.VisualStudio.TestPlatform.ObjectModel.dll' has already loaded from a different location. It cannot be loaded from a new location within the same appdomain.
Warning: Could not load file or assembly 'file:///C:\Users\xxxxxx\AppData\Local\Temp\VSTFSBuild\cbce9143-7133-45af-a385-0c25ab072cef\76770201\phantomjs.exe' or one of its dependencies. The module was expected to contain an assembly manifest.
Jun 24, 2013 at 12:41 PM
Edited Jun 24, 2013 at 12:42 PM
Hi,

Please let me know "How to integrate in Team Build 2010" for Qunit via Chutzpah 2.4.1 Version.

i saw the Javascript link (http://blogs.msdn.com/b/visualstudioalm/archive/2012/07/09/javascript-unit-tests-on-team-foundation-service-with-chutzpah.aspx?CommentPosted=true#commentmessage).

Someone please tell me proper steps to integrate With Team Build 2010 to run the Q-Unit with Code Coverage.

Thanks in Adavance.
Jun 27, 2013 at 5:59 PM
Same exact issue as jimsleon. Visual Studio 2012.2 and TFS 2012 on premise. Digging into the log I see the following:

C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe /nologo /usestderr /searchpathroot:"C:\Builds\theAppPath\Binaries" /resultsfileroot:"C:\Builds\theAppPath\TestResults" /testcontainer:"C:\Builds\theAppPath\Binaries_PublishedWebsites\DIB.ItemManagementSystem.Web\bin\jasmine-serviceWebTests.js" /testcontainer:"C:\Builds\theAppPath\Binaries_PublishedWebsites\DIB.ItemManagementSystem.Web\Scripts\knockout\knockout.namespaces-latest.debug.js" /testcontainer:"C:\Builds\theAppPath\Binaries_PublishedWebsites\DIB.ItemManagementSystem.Web\Tests\jasmine-serviceWebTests.js" /testcontainer:"C:\Builds\theAppPath\Binaries_PublishedWebsites\DIB.ItemManagementSystem.Web\Tests\TestingSystemTest.js" /testcontainer:"C:\Builds\theAppPath\Binaries\Tests\jasmine-serviceWebTests.js" /publish:"http://ntserv89:8080/tfs/defaultcollection" /noprompt /publishbuild:"vstfs:///Build/Build/4958" /teamproject:"DIB" /platform:"Any CPU" /flavor:"Debug" Loading C:\Builds\theAppPath\Binaries\_PublishedWebsites\AWebProject\bin\jasmine-serviceWebTests.js...
C:\Builds\theAppPath\Binaries_PublishedWebsites\AWebProject\bin\jasmine-serviceWebTests.js
File extension specified '.js' is not a valid test extension.
Coordinator
Jun 28, 2013 at 4:49 AM
I have not tested the tfs build using Chutzpah with code coverage. When I have time I will experiment with that and get back to you.
Aug 7, 2013 at 10:50 PM
@mmanela
I believe the chutzpah need to implement custom code coverage api needed in .runsettings file.

Configuring Unit Tests by using a .runsettings File
Customizing Code Coverage Analysis

There a issue for this ?
Sep 13, 2013 at 4:24 PM
Edited Sep 13, 2013 at 4:26 PM
sachbist wrote:
Hi,

Please let me know "How to integrate in Team Build 2010" for Qunit via Chutzpah 2.4.1 Version.

i saw the Javascript link (http://blogs.msdn.com/b/visualstudioalm/archive/2012/07/09/javascript-unit-tests-on-team-foundation-service-with-chutzpah.aspx?CommentPosted=true#commentmessage).

Someone please tell me proper steps to integrate With Team Build 2010 to run the Q-Unit with Code Coverage.

Thanks in Adavance.
When you follow the steps their, do you got the pass/fail results for your JS unit tests or not?
Believe me, there is no a proper method to get Code Coverage under TFS build servers for js or html.
Is there any idea how to get the code coverage as .coverage file for .js or *.html file?
For the previous problem, When I Edit any build definition, especially when I click on "Process" tab I got this msg in "Output" window:
Summary: There were 0 failures, 2 errors and 1 warnings loading custom activities and services.
Error: Method 'ToXml' in type 'Chutzpah.VS2012.TestAdapter.ChutzpahAdapterSettings' from assembly 'Chutzpah.VS2012.TestAdapter, Version=2.2.1.173, Culture=neutral, PublicKeyToken=1ca802c37ffe1896' does not have an implementation.
Error: API restriction: The assembly 'file:///C:\Users...\AppData\Local\Temp\VSTFSBuild\283ed5fc-170f-4a1c-b4f7-bddc9fd18de1\680756381\Microsoft.VisualStudio.TestPlatform.ObjectModel.dll' has already loaded from a different location. It cannot be loaded from a new location within the same appdomain.
Warning: Could not load file or assembly 'file:///C:\Users...\AppData\Local\Temp\VSTFSBuild\283ed5fc-170f-4a1c-b4f7-bddc9fd18de1\680756381\phantomjs.exe' or one of its dependencies. The module was expected to contain an assembly manifest.
I went to this path:
C:\Users...\AppData\Local\Temp\VSTFSBuild\
and deleted everything there but for vain.
Well, I'm not in a harry to find the solution because all my JS and html unit tests are being run perfectly under TFS 2012 Build servers, still though, I'm so eager to find a solution for this and stop bugging me every time I edit a build definition config.
Nov 25, 2013 at 8:19 PM
Don't have time to chat, but did want to mention, I got Chutzpah to work on TFS on premises. The Test runner was stuck on MSTest. To get the Visual Studio Test runner into my build template I had to add that section: [http://blogs.msdn.com/b/biranchip/archive/2013/05/29/visual-studio-test-runner-not-an-option-for-tfs-2012-build-definition.aspx](MSDN Blog post on how to do that.)

After setting that up, .js tests would run. yay.

I am now trying to figure out how to put my tests into their own project. It looks like it has to be a website project, otherwise Chutzpah doesn't receive the js files in the outputs directory. I'd like to be able to get a test project to render a tests folder somewhere in the output directory that would keep proper paths to the sites within _Publishedwebsites, but test projects apparently don't render to the output directory.
Dec 5, 2013 at 1:28 PM
Edited Dec 5, 2013 at 1:29 PM
I double checked our build definition, and they are already running using the Visual Studio Test Runner. Unfortunately, that did not solve my problem. Thanks for the post though.
Apr 7 at 12:24 PM
Edited Apr 7 at 12:24 PM
Hi there.

I want to up this topic cos I have same problem.

My configuration is VS 2013 and lates version of Chutzpah (i.e. 3.2.1).

I have downloaded the plugin, extracted all files and added them to repository, added path to the Build Controller.

When I try to edit "Process" section on my build definition I have an error:

Error: Method 'ToXml' in type 'Chutzpah.VS2012.TestAdapter.ChutzpahAdapterSettings' from assembly 'Chutzpah.VS2012.TestAdapter, Version=3.2.1.0, Culture=neutral, PublicKeyToken=1ca802c37ffe1896' does not have an implementation.

I'm using tfs.visualstudio.com to build my project.

Any ideas?
Coordinator
Apr 7 at 2:34 PM
Wierd. Many people have seen this but I have not been able to repro. I have a test project on .visualstudio.com that works fine. Could you create a project with a repro and share it with me?
Apr 7 at 8:20 PM
Edited Apr 7 at 8:22 PM
I have created simple project. Same error. And some new.

First was about "Microsoft.VisualStudio.TestPlatform.ObjectModel.dll".
I have added this file to repo too.
Now I have this:
_Error: Could not load file or assembly 'Microsoft.VisualStudio.Shell.11.0, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
Error: Could not load file or assembly 'file:///C:\Users\xxx\AppData\Local\Temp\VSTFSBuild\c14bef24-5df3-4352-93bf-d7e623bfbf4b\A521245264\Microsoft.VisualStudio.Shell.11.0.dll' or one of its dependencies. The system cannot find the file specified.

Error: Method 'ToXml' in type 'Chutzpah.VS2012.TestAdapter.ChutzpahAdapterSettings' from assembly 'Chutzpah.VS2012.TestAdapter, Version=3.2.1.0, Culture=neutral, PublicKeyToken=1ca802c37ffe1896' does not have an implementation.

Warning: Could not load file or assembly 'file:///C:\Users\xxx\AppData\Local\Temp\VSTFSBuild\c14bef24-5df3-4352-93bf-d7e623bfbf4b\A521245264\phantomjs.exe' or one of its dependencies. The module was expected to contain an assembly manife_st.
I have only VS 2013 on my home PC.

On work PC I still have error from first post. VS 2012 and 2013 installed.
Apr 7 at 11:04 PM
Just a thought: could this be because you have the VS SDK installed and they do not?
Coordinator
May 17 at 3:24 PM
@Gengzu, I do repro the error when I go to the process tab when using your repro for the VSOnline account. But I am able to queue a build and have it run the tests. Is that consistent with what you are seeing?
Jun 4 at 1:50 PM
Has anyone run across the following errors when using TFS 2013?
Error: System.ArgumentNullException: Value cannot be null.
Parameter name: path
   at System.IO.File.ReadAllText(String path)
   at Chutzpah.Wrappers.FileSystemWrapper.<>c__DisplayClass1.<GetText>b__0()
   at Chutzpah.Wrappers.FileSystemWrapper.RetryT
   at Chutzpah.Wrappers.FileSystemWrapper.GetText(String path)
   at Chutzpah.TestHarnessBuilder.CreateTestHarness(TestContext testContext, TestOptions options)
   at Chutzpah.TestRunner.<>c__DisplayClass3.<ExecuteTestContexts>b__2(TestContext testContext)
While Running:C:\Builds\1\AgMapping\AgMappingDevelop\bin\js\SimpleTest.js
I've removed all our tests but one simple one and still get the error:
/// <reference path="libs/jasmine.js" />
/*
 * SIMPLE TEST
 */
describe("JS-> SIMPLE TEST", function () {
  it('shouldDoSomething', function () {
    var simpleCheck;
    expect('simpleCheck').toBeDefined();
  });
});
Also get the same error for all the angular.js files we use in our project.

We are using TFS 2013 and the latest Chutzpah (3.2.2). All our tests are using Jasmine and have the JSRunners and TestFiles all referenced on the build server. All the tests run successfully on our local machines.

Thanks,
Chuck