I am currently working on some .NET-based software (.NET Framework 3.5 SP1) that integrates with HP Quality Center 10.0 through the COM Client API (often called TDApiOle80 or TDApiOle80.TDConnection).
We use XUnit 1.6.1.1521 and Gallio 3.1.397.0 (called from the msbuild file)
We go through the process:
- Create connection
- Test run
- Closing connection
- Positioning
- Forcing GC.Collection () / GC.AwaitingPendingFinalizers ()
For each integration test - and each integration test is performed with the Fact timeout set in it.
The problem is that it appears after several tests (say, around 10). The quality center blocks endlessly on a call - and the whole Gallio freezes and will no longer respond.
We initially discovered that xunit.net only applied a timeout to the code inside this fact - so it will wait indefinitely for the constructor or implement methods to complete - so we moved this logic to the body of the tests just for confirmation ... but it doesn’t solved the problem (it will still freeze after running a certain number of tests).
The same thing happens when using TestDriven.Net - 1 or more tests can be run interactively, but more than 10 tests and the whole run freezes - and our only choice is to kill the ProcessInvocation86.exe process used by TD. Net.
Does anyone have any tips / advice on how to stop this all together, or at least isolate my integration tests from such problems - so that the tests in which the QC API is blocked indefinitely, time out and allow Gallio proceed to the next test.
Update
A hint about using the STA thread helped move the problem a bit - with the XUnit.Net custom attribute, we now run the test in our own STA thread. This completely stopped Gallio / TestDriven.Net, so we can enable the integration tests on our hudson build server.
public class StaThreadFactAttribute : FactAttribute { const int DefaultTime = 30000; // 30 seconds public StaThreadFactAttribute() { Timeout = DefaultTime; } protected override System.Collections.Generic.IEnumerable<Xunit.Sdk.ITestCommand> EnumerateTestCommands(Xunit.Sdk.IMethodInfo method) { int timeout = Timeout; Timeout = 0; var commands = base.EnumerateTestCommands(method).ToList(); Timeout = timeout; return commands.Select(command => new StaThreadTimeoutCommand(command, Timeout, method)).Cast<ITestCommand>(); } } public class StaThreadTimeoutCommand : DelegatingTestCommand { readonly int _timeout; readonly IMethodInfo _testMethod; public StaThreadTimeoutCommand(ITestCommand innerComand, int timeout, IMethodInfo testMethod) : base(innerComand) { _timeout = timeout; _testMethod = testMethod; } public override MethodResult Execute(object testClass) { MethodResult result = null; ThreadStart work = delegate { try { result = InnerCommand.Execute(testClass); var disposable = testClass as IDisposable; if (disposable != null) disposable.Dispose(); } catch (Exception ex) { result = new FailedResult(_testMethod, ex, this.DisplayName); } }; var thread = new Thread(work); thread.SetApartmentState(ApartmentState.STA); //Set the thread to STA thread.Start(); if (!thread.Join(_timeout)) { return new FailedResult(_testMethod, new Xunit.Sdk.TimeoutException((long)_timeout), base.DisplayName); } return result; } }
Instead, we now see a conclusion similar to this when running tests using TestDriven.Net - by the way, running the same package several times will result in either tests passing, or, as a rule, in one or two test failures. And after the first failure, the second failure leads to the error "Error unloading appdomain".
Test 'IntegrationTests.Execute_Test1' failed: test run time exceeded: 30000ms
Test 'T: IntegrationTests.Execute_Test2' Error: Error unloading AppDomain. (Exception from HRESULT: 0x80131015) System.CannotUnloadAppDomainException: Error unloading appdomain. (Exception from HRESULT: 0x80131015) in System.AppDomain.Unload (AppDomain domain) in Xunit.ExecutorWrapper.Dispose () in Xunit.Runner.TdNet.TdNetRunner.TestDriven.Framework.ITestRunner.RunMember (ITestListener) listener TestDriven.TestRunner.AdaptorTestRunner.Run (ITestListener testListener, ITraceListener traceListener, String assemblyPath, String testPath) in TestDriven.TestRunner.ThreadTestRunner.Runner.Run ()
4 passed, 2 failed, 0 missed, took 50.42 seconds (xunit).
I don’t know yet why the Quality Center API hangs indefinitely in random order - it will investigate in the near future.
Update 07/27/2010
I finally found the cause of the hang - here is the problem code:
connection = new TDConnection(); connection.InitConnectionEx(credentials.Host); connection.Login(credentials.User, credentials.Password); connection.Connect(credentials.Domain, credentials.Project); connection.ConnectProjectEx(credentials.Domain, credentials.Project, credentials.User, credentials.Password);
It looks like the Connect call, followed by ConnectProjectEx, has a chance to block (but it is not deterministic). Removing redundant connection calls seems to significantly increase test stability - the correct connection code:
connection = new TDConnection(); connection.InitConnectionEx(credentials.Host); connection.ConnectProjectEx(credentials.Domain, credentials.Project, credentials.User, credentials.Password);
Inheriting the code base, I did not give the communication code, which I thought a lot.
One thing I still have to figure out is that even with the above timeout Thread.Join (timeout) never returns. You can attach a debugger, and it just shows that the test thread is in a connection / wait operation. Perhaps something is happening with the execution in the STA thread?