Speed ​​up msbuild with fsutil hardlink create instead of slow copy tasks

Question

I am trying to speed up the build (csharp, msbuild, .net 3.5). Replace the copy with fsutil hardlink create.

Before, I almost got it with running a script on sln files and making dll private = false links, and then when the post build event created hard links. The problem is that transitive dependencies are not included. Therefore, I think I need to refer to the ResolveAssemblyReference in msbuild to get the transitive dependencies that I need for hard binding.

Any ideas?

This person tried the same, but did not publish the final decision.

To be clear: I want to keep separate bin directories, but instead of copying the file from one to another to create a hard link from the source (link or dependency) to the destination (current project garbage). This is much faster and gives about the same effect as copying.

Answer 1

This is supported in VS 2010. But not 2008. See the UseHardLinksIfPossible parameter for copying to _CopyFilesMarkedCopyLocal.

See also http://social.msdn.microsoft.com/Forums/en/tfsbuild/thread/9382a3d8-4632-4826-ad15-d5e845080981 , http://msdn.microsoft.com/en-us/library/ms171466 (v = VS.90) .aspx for context.

Undo target _CopyFilesMarkedCopyLocal. We add something like this to the csproj files below: <Import Project="..\..\..\..\..\\CommonBuild\TW.Override.Microsoft.Common.targets" /> (This is automatic is added to the file using the nant task for each complete nant build, so each project is profitable).

Our new goals file:

  <Project ToolsVersion = "3.5" DefaultTargets = "Build" xmlns = "http://schemas.microsoft.com/developer/msbuild/2003">
      <UsingTask TaskName = "CopyWithHardlinkOption" AssemblyFile = ".. \ lib \ TWBuildOptimization \ TW.Hardlinker.dll" />
   <! -
     =================================================== ===========
                                         _CopyFilesMarkedCopyLocal
     Overridden in order to allow hardlinking with our custom Copy Task.                                         

     Hardlinking is a major performance improvement.  Sometimes 50% of the time compared to copying.

     Copy references that are marked as "CopyLocal" and their dependencies, including .pdbs, .xmls and satellites.
     =================================================== ===========
   ->
     <Target
         Name = "_ CopyFilesMarkedCopyLocal">
         <CopyWithHardlinkOption
             SourceFiles = "@ (ReferenceCopyLocalPaths)"
             DestinationFiles = "@ (ReferenceCopyLocalPaths -> '$ (OutDir)% (DestinationSubDirectory)% (Filename)% (Extension)')"
             SkipUnchangedFiles = "true"
               UseHardlinksIfPossible = "true"
             OverwriteReadOnlyFiles = "$ (OverwriteReadOnlyFiles)">
             <Output TaskParameter = "DestinationFiles" ItemName = "FileWritesShareable" />
         </CopyWithHardlinkOption>
     </Target>
 </Project>

See the msbuild 4 Copy UseHardlinksIfPossible option. I supported this up to 3.5 by decompiling and overriding. Relevant logic in CopyFileWithLogging:

  // The port from 4.0 task that allows hardlinking bool hardlinkSucceeded = false; if (UseHardlinksIfPossible) { if (File.Exists(destinationFile)) { FileUtilities.DeleteNoThrow(destinationFile); } if (!TwNativeMethods.CreateHardLink(destinationFile, sourceFile, IntPtr.Zero)) { var win32Exception = new Win32Exception(Marshal.GetLastWin32Error()); Log.LogMessage(MessageImportance.High, "Hardlinking had a problem {0}, will retry copying. {1}", new object[] {win32Exception.Message, win32Exception}); } hardlinkSucceeded = true; } if (!hardlinkSucceeded) { Log.LogMessageFromResources(MessageImportance.Normal, "Copy.FileComment", new object[] { sourceFile, destinationFile }); Log.LogMessageFromResources(MessageImportance.Low, "Shared.ExecCommand", new object[0]); Log.LogCommandLine(MessageImportance.Low, "copy /y \"" + sourceFile + "\" \"" + destinationFile + "\""); File.Copy(sourceFile, destinationFile, true); } // end port 

It was also necessary to add the following data:

 // decompiled from 4.0 internal static class TwNativeMethods { [DllImport("kernel32.dll", CharSet=CharSet.Unicode, SetLastError=true)] internal static extern bool CreateHardLink(string newFileName, string exitingFileName, IntPtr securityAttributes); } // decompiled from 4.0 internal static class FileUtilities { internal static void DeleteNoThrow(string path) { try { File.Delete(path); } catch (Exception exception) { if (ExceptionHandling.NotExpectedException(exception)) { throw; } } } } // decompiled from 4.0 internal static class ExceptionHandling { // Methods internal static bool IsCriticalException(Exception e) { return (((e is StackOverflowException) || (e is OutOfMemoryException)) || ((e is ExecutionEngineException) || (e is AccessViolationException))); } internal static bool NotExpectedException(Exception e) { return (((!(e is UnauthorizedAccessException) && !(e is ArgumentNullException)) && (!(e is PathTooLongException) && !(e is DirectoryNotFoundException))) && ((!(e is NotSupportedException) && !(e is ArgumentException)) && (!(e is SecurityException) && !(e is IOException)))); } internal static bool NotExpectedReflectionException(Exception e) { return ((((!(e is TypeLoadException) && !(e is MethodAccessException)) && (!(e is MissingMethodException) && !(e is MemberAccessException))) && ((!(e is BadImageFormatException) && !(e is ReflectionTypeLoadException)) && (!(e is CustomAttributeFormatException) && !(e is TargetParameterCountException)))) && (((!(e is InvalidCastException) && !(e is AmbiguousMatchException)) && (!(e is InvalidFilterCriteriaException) && !(e is TargetException))) && (!(e is MissingFieldException) && NotExpectedException(e)))); } } 

Answer 2

Have you tried to define your own target that starts after the ResolveAssemblyReferences target? After completing the ResolveAssemblyReferences, you can use the @ (ReferencePath) element to control the creation of links. Think about implementing AfterResolveReferences, which in Microsoft.Common.targets is just an empty place for you to redefine.