Using Robocopy to Deploy Sites

Question

I want to be able to quickly deploy updates to a rather busy site. For small sites, I would just FTP new files over old ones. This, however, has several large dlls that are regularly updated, and when they are copied, the site is effectively inaccessible (plus there are efforts to back them up in case something goes wrong.

My plan is to use TortoiseHg to synchronize with an intermediate copy on the server via FTP (using netdrive or something similar). I can verify that everything works smoothly, and as soon as this is completed, I would like to run a .bat file (or something else) that will create a backup of the live site (preferably only the files that are going to change, but that’s not critical), and then copy the newly modified files to the site in real time.

If possible, I also want the copy to ignore certain directories (for example, user downloads) so that it does not overwrite these files on the site in real time?

I heard RoboCopy is the way to go, but I'm not sure where to start. Would I need to invoke 2 commands (1 for the initial backup and one for the copy)? Is there a way to restore a site in real time? Previous state should something go wrong?

The site is located in ASP.NET and will be copied to the Windows 2003 server.

EDIT: It becomes a little complicated when the web.config elements have changed and they need to be combined so that the settings of the intermediate servers (application settings, connection strings, etc.) are not deployed to a real site. How is this handled?

Answer 1

We use the following

• first create a website using msbuild at cruisecontrol.net to create binaries.

• Archive files in expanded form in a folder with timestamps to avoid data loss in case of a problem.

  C:\DevTools\Robocopy\robocopy.exe /R:1 /W:10 /mir "D:\WebSite\Files" "D:\Webarchive\ArchivedFiles\Documents.%date:~0,-8%.%date:~3,-5%.%date:~6%.%time:~0,-9%.%time:~3,-6%.%time:~6,-3%" /XF *.scc

• stop website

• Deploy the website by copying everything except the files we archived (/ XD - eXclude directory)

  C:\DevTools\Robocopy\robocopy.exe /R:1 /W:10 /mir "c:\dev\site" "D:\WebSite" /XF *.scc /XD "D:\WebSite\Files"

• copy and rename (with xcopy, this time) release.config with the correct information in d: \ Website \ web.config (in fact, what we did before, now we have a homebrew conversion mechanism to change parts of dev web.config on the fly).

• restart the website.
• (optional) delete the archive you made in step 2

In your case, you will need to add the / XD flags for any directory that you want to ignore, for example, when loading users. And if the production web.config file isn’t complicated, I would recommend just copying the release.config file that you support as part of the project, next to web.config

Answer 2

Is Robocopy a strict requirement? Why not use MSBuild? Everything that you specified can be done painlessly in MSBuild.

 <!-- Attempt to build new code --> <MSBuild Projects="$(BuildRootPath)\ThePhotoProject.sln" Properties="Configuration=$(Environment);WebProjectOutputDir=$(OutputFolder);OutDir=$(WebProjectOutputDir)\" /> <!-- Get temp file references --> <PropertyGroup> <TempConfigFile>$([System.IO.Path]::GetTempFileName())</TempConfigFile> <TempEnvironmentFile>$([System.IO.Path]::GetTempFileName())</TempEnvironmentFile> </PropertyGroup> <!-- Copy current web configs to temp files --> <Copy SourceFiles="$(OutputFolder)\web.config" DestinationFiles="$(TempConfigFile)"></Copy> <Copy SourceFiles="$(OutputFolder)\web.$(Environment).config" DestinationFiles="$(TempEnvironmentFile)"></Copy> <ItemGroup> <DeleteConfigs Include="$(OutputFolder)\*.config" /> </ItemGroup> <Delete Files="@(DeleteConfigs)" /> 

...

 <!-- Copy app_offline file --> <Copy SourceFiles="$(CCNetWorkingDirectory)\Builder\app_offline.htm" DestinationFiles="$(DeployPath)\app_offline.htm" Condition="Exists('$(CCNetWorkingDirectory)\Builder\app_offline.htm')" /> <ItemGroup> <DeleteExisting Include="$(DeployPath)\**\*.*" Exclude="$(DeployPath)\app_offline.htm" /> </ItemGroup> <!-- Delete Existing files from site --> <Delete Files="@(DeleteExisting)" /> <ItemGroup> <DeployFiles Include="$(OutputFolder)\**\*.*" /> </ItemGroup> <!-- Deploy new files to deployment folder. --> <Copy SourceFiles="@(DeployFiles)" DestinationFiles="@(DeployFiles->'$(DeployPath)\%(RecursiveDir)%(Filename)%(Extension)')" /> <!-- Delete app_offline file --> <Delete Files="$(DeployPath)\app_offline.htm" Condition="Exists('$(DeployPath)\app_offline.htm')" /> 

Answer 3

On Nix-based servers, I would like to use Rsync and I understand that on Windows, you can use DeltaCopy which is Rsync port and open sources (never used DeltaCopy, so please study it carefully). In any case, if it works as RSYNC, it quickly and only updates files that have been changed.

You can use various configuration options to delete files on a target that were deleted from the source, and you can also use an add-in in a file that will exclude files or directories, i.e. a local config, you do not want to copy. etc.

You should be able to collapse all of this into a single script to run when necessary, which means that you can test and time so that you know what is happening.

Answer 4

Check out these links to see if they help:

• ASP.NET Website Continuous Integration + Deployment Using CruiseControl.NET, Subversion, MSBuild, and Robocopy
• Multiple Folder Deployment Using Robocopy

You will find that robocopy.exe /? extremely helpful. In particular, you will need the /XF switch to exclude files and /XD to exclude folders.

You will need to write a script (e.g. bat, powershell, cscript) to take care of problems with web.config.

Answer 5

Microsoft itself uses robocopy to deploy updates to some sites.

I do not know if you have several servers, but our deployment script went something like: 1) Stop IIS (which would be the server from the load balancing rotation, 2) RoboCopy / MIR from \ Staging \ path \ to \ webroot to \ WEB ## \ path \ to \ webroot, where ## is the server number, 3) Launch IIS. This was done after the site passed a smoke test on an intermediate server.

This does not help much in your configuration problem, but our configuration files for installation and production were the same.

Answer 6

What you need (and I need) is to synchronize the program with the ability to back up files on the server and make quick copies on top of ftp files on them, probably by copying them first in a temporary directory or by partial update.

This is one of the programs I found: http://www.superflexible.com/ftp.htm

Answer 7

WebDeploy is a much better way to handle deployments (see Scott H http://www.hanselman.com/blog/WebDeploymentMadeAwesomeIfYoureUsingXCopyYoureDoingItWrong.aspx )

But Robocopy is a great low-cost deployment tool that I still use on some sites (I didn’t take the time to change them to webdeploy). Robocopy is similar to xcopy, but with a much richer set of options. Thus, you will need 2 Robocopy commands (1 for backup and 1 for deployment). Usually I make a backup command when files are put.

Managing configuration files is always difficult (and a big reason to use webdeploy). One approach is to keep a copy of the configuration files for each environment checked in your source control (e.g. web.dev.config, web.uat.config, web.prod.config, etc.). Holding (or deploying a script) will capture and rename the necessary configuration file.

Answer 8

You will probably need to use a combination of tools.

I would look at DFSR (file server role) with a read-only folder on your live site (this is one-way replication).

It is very easy to configure, have a convenient graphical interface, the ability to exclude files based on location and / or masks, and with shadow copying enabled, you can run it in the schedules you set and update only those files that change (or run it on a schedule or even run it manually). The beauty of this is that when it is set up, you no longer need to touch it.

After you have most of your file replicas, you can get help automating a possible merge on web.config, assuming you want to automate.

Answer 9

MSBuild is great, with the exception of one minor (or large, depending on your point of view) error. It restores binary files every time you run assembly. This means that for deployments from TEST to PRODUCTION or STAGE to PRODUCTION (or regardless of what you call in preliminary mode), if you use MSBuild, you do not push existing binaries from one environment to another, you recreate them. This also means that you are sure that NOTHING has changed in the source code repository since you did MSBuild in your pre-production environment. Allowing even the slightest chance of changing anything, significant or minor, means that you will not promote a fully tested product in your production environment. In the places where I work, this is an unacceptable risk.

Enter Robocopy. With Robocopy, you copy (hopefully) a fully tested product into your production environment. Then you will need to manually modify your web.config / app.config to reflect the production environment, or use the conversion tool to do this. For this purpose I used the "Configuration Transformation Tool" available for SourceForge - it works the same way as the MSBuild web / app.config conversions.