Tests on Visual Studio test slower on TFS Build

Question

In my project, there are more than 1000 unit tests that run on the local computer in less than 10 seconds. But when they run on TFS Build, some tests run much slower than others. 3 of them work after about 1-2 minutes, the other 4 after 5-30 seconds, and the rest in a split second. I noticed that all those slow tests use fakes from Microsoft Fakes, and each of them is the first to run a class in it. But many other tests also use fakes (even more intensively) and run in normal time. I would like to know what can slow down and how I can fix it.

Edit: I noticed that every slow test passes after the test without makeup. Perhaps this slowdown is caused by the initialization of the ShimsContext. In my test classes, a ShimsContext is created and hosted in the TestInitialize and TestCleanup methods. Does this affect performance significantly?