Microsoft Fakes & TFS 2012 Visual Studio Test Runner - unit test unsuccessfully sporadically

Question

Environment

• Visual Studio 2012 Premium 3 Update
• Team Foundation Server 2012 Update 3
• edit: .NET Framework 4
• DefaultTemplate (DefaultTemplate.11.1.xaml)
  • Visual Studio Test Drive
    
    • Target platform: X86

Problem

I have 2 Unit-Test-Assemblies:

• ExtensionTests
• UserTests

both use the Microsoft Fakes Framework (using Stubs and Shims).

Running unit tests locally works fine (tested on 4 different computers, even on Visual Studio installed on the build server), however, if we create using the build agent, some unit tests do not work except, for example:

Unable to create instance of class UserTests.ClientUserTest. Error: System.TypeLoadException: Could not load type 'WorldDirect.CCM.Shared.Backend.SmartClassic.Fakes.StubClient' from assembly 'WorldDirect.Smart.Backend.Fakes, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. 

or the same for laying:

 Test method ExtensionTests.ExtensionTests.UpdateExtensionValidate_NoGrnp_ChecksIpPbxDependencies threw exception: System.TypeLoadExceptio: Could not load type 'WorldDirect.CCM.Shared.Backend.SmartClassic.Fakes.ShimIpPbxRemoteDestination' from assembly 'WorldDirect.Smart.Backend.Fakes, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. 

Interestingly, they fail sporadically; I tried to compile some statistics:

  # | Outcome | Comment 19 | 64/64 passed | only 'UserTests'-UnitTests run, 'ExtensionTests' inactive 20 | 37/64 passed | same source code as #19 21 | 64/64 passed | same source code as #19 22 | 64/66 passed | all 'UserTests' run as well as 2 'ExtensionTests', others still inactive; the 2 ExtensionTests failed with ShimIpPbxRemoteDestination-TypeLoadException 23 | 38/65 passed | same source code as #22, notice NOTHING has changed, however 1 unit test was not even run in 'ExtensionTests'; however the second succeeds; all unit tests in 'UserTests' using MS Fakes fail with a StubClient-TypeLoadException 24 | 38/65 passed | same source code as #22 new day - no more luck :-( 1 | 37/64 passed | uncommented the 2 'ExtensionTests', so same code as #19 2 | 37/64 passed | so same code as #1 3 | 64/64 passed | so same code as #1; suddenly they all work again 

Please note that in addition to these changes, no one has changed the source code.
All assemblies were made using a verbose log level; Shim Diagnostic = "true",
only explicitly needed gaskets / stubs are included in the .fakes file (otherwhise MSBUILD failed using exit code 1).

All this seems to me completely magical, maybe someone already had the same problems, or someone has a hint.

Thanks in advance

Answer 1

I think I know where the problem is, the TypeNotFoundException exception was correct, at least the corresponding types were not thrown (as shown by Reflector), it made me think. More precisely, only one generated fake file generated per assembly is located under [buildName] \ Binaries-, and since both my assemblies are fabricated from the same Assembly (backend.dll) - this most likely happened in a race state - the only assembly which was built later than the other, with its final gaskets / plugs; this does not clarify all the problems.

In any case, the reason this does not appear in a regular VS assembly is because assemblies are created in their corresponding [assembly] \ FakeAssemblies directory without affecting any other unit test (as it should be).

So my idea to get around this problem was to put ALL the gaskets / stubs used by both test builds in both .fakes-configuration-files; the reflector showed that this time all the necessary gaskets / plugs were really created, however, the assembly failed with

 Exception Message: MSBuild error 1 has ended this build. 

Without additional information in any (detailed) journal. This problem is caused by not using the generated shim / stub object specified in the .fakes file (and, of course, this is also the problem just shown by the build server, it will be too boring otherwise ;-)).

So, the last solution I came across is to put ALL code that uses gaskets / stubs in the same assembly, where only 1 .fakes file sets up a fake assembly.

Now it works fine :-), but I think that MS-Dev-Team can still do something, have not tested it with TFS2013; going to send an error anyway (as I already opened 2)

Hope this helps someone with the same issue.