Upload BIG files via HTTP

Question

I am trying to upload really large VM images (size 5-15 GB) to an HTTP server using PowerShell.

I tried using for several methods (here script links with net.WebClient.UploadFile and script with Invoke-webRequest )

It works great for files smaller than 2 GB, but not for files larger than this.

I am trying to work with httpWebRequest directly, but I cannot insert a FileStream into it.

So my question is: how to put filestream in webrequest?

Or in a more general sense: how to upload a huge file via http using PowerShell?

 $Timeout=10000000; $fileName = "0.iso"; $data = "C:\\$fileName"; $url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName"; #$buffer = [System.IO.File]::Open("$data",[System.IO.Filemode]::Open, [System.IO.FileAccess]::Read) #Err Cannot convert argument "buffer", with value: "System.IO.FileStream", for "Write" to type "System.Byte[]": #$buffer = gc -en byte $data # too much space in memory $buffer = [System.IO.File]::ReadAllBytes($data) #Limit 2gb [System.Net.HttpWebRequest] $webRequest = [System.Net.WebRequest]::Create($url) $webRequest.Timeout = $timeout $webRequest.Method = "POST" $webRequest.ContentType = "application/data" #$webRequest.ContentLength = $buffer.Length; $webRequest.Credentials = New-Object System.Net.NetworkCredential("admin", "admin123"); $requestStream = $webRequest.GetRequestStream() $requestStream.Write($buffer, 0, $buffer.Length) $requestStream.Flush() $requestStream.Close() [System.Net.HttpWebResponse] $webResponse = $webRequest.GetResponse() $streamReader = New-Object System.IO.StreamReader($webResponse.GetResponseStream()) $result = $streamReader.ReadToEnd() return $result $stream.Close() 

Answer 1

By default, HttpWebRequest buffers data in memory. Just set the HttpWebRequest.AllowWriteStreamBuffering property to false and you can upload files of almost any size. See msdn for more details.

Answer 2

Thanks @Stoune, the last thing that helped to get a final working solution.

In addition, it is necessary to organize the reading and writing of the file stream to the webrequest buffer. And this may be due to this piece of code:

 $requestStream = $webRequest.GetRequestStream() $fileStream = [System.IO.File]::OpenRead($file) $chunk = New-Object byte[] $bufSize while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) ) { $requestStream.write($chunk, 0, $bytesRead) $requestStream.Flush() } 

And the final script looks like this:

 $user = "admin" $pass = "admin123" $dir = "C:\Virtual Hard Disks" $fileName = "win2012r2std.vhdx" $file = "$dir/$fileName" $url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName" $Timeout=10000000 $bufSize=10000 $cred = New-Object System.Net.NetworkCredential($user, $pass) $webRequest = [System.Net.HttpWebRequest]::Create($url) $webRequest.Timeout = $timeout $webRequest.Method = "POST" $webRequest.ContentType = "application/data" $webRequest.AllowWriteStreamBuffering=$false $webRequest.SendChunked=$true # needed by previous line $webRequest.Credentials = $cred $requestStream = $webRequest.GetRequestStream() $fileStream = [System.IO.File]::OpenRead($file) $chunk = New-Object byte[] $bufSize while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) ) { $requestStream.write($chunk, 0, $bytesRead) $requestStream.Flush() } $responceStream = $webRequest.getresponse() #$status = $webRequest.statuscode $FileStream.Close() $requestStream.Close() $responceStream.Close() $responceStream $responceStream.GetResponseHeader("Content-Length") $responceStream.StatusCode #$status 

Answer 3

To download to Sonatype Nexus3, I used the code below. It took a little time to figure this out by working with the Nexus3 download and response, as well as downloading and uploading large files (over 2 GB). We have Apache in front of Nexus3, which takes care of https connections. Using basic authentication, make sure Nexus responds correctly when Apache is in front of it. Sending preauthentication using HEAD and using a corrected download for large files fixed that downloading large files did not end prematurely.

Uploading large files through Invoke-WebRequest is also not error-prone. Now I use WebRequest via .Net and it works, see Download-File .

And when the code was run through an automated process (from System Center Orchestrator), https will fail. Therefore, we force TLS 1.2 to detect the https scheme.

 function New-HttpWebRequest { <# .SYNOPSIS Creates a new [System.Net.HttpWebRequest] ready for file transmission. .DESCRIPTION Creates a new [System.Net.HttpWebRequest] ready for file transmission. The method will be Put. If the filesize is larger than the buffersize, the HttpWebRequest will be configured for chunked transfer. .PARAMETER Url Url to connect to. .PARAMETER Credential Credential for authentication at the Url resource. .EXAMPLE An example #> param( [Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Url, [Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()] [pscredential]$Credential, [Parameter(Mandatory=$true)] [long]$FileSize, [Parameter(Mandatory=$true)] [long]$BufferSize ) $webRequest = [System.Net.HttpWebRequest]::Create($Url) $webRequest.Timeout = 600 * 1000; $webRequest.ReadWriteTimeout = 600 * 1000; $webRequest.ProtocolVersion = [System.Net.HttpVersion]::Version11; $webRequest.Method = "PUT"; $webRequest.ContentType = "application/octet-stream"; $webRequest.KeepAlive = $true; $webRequest.UserAgent = "<I use a specific UserAgent>"; #$webRequest.UserAgent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'; $webRequest.PreAuthenticate = $true; $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password)); $webRequest.Headers["Authorization"] = "Basic $auth" if (Get-UseChunkedUpload -FileSize $FileSize -BufferSize $BufferSize) { Write-Verbose "FileSize is greater than BufferSize, using chunked transfer."; $webRequest.AllowWriteStreamBuffering = $false; $webRequest.SendChunked = $true; } else { # Filesize is equal to or smaller than the BufferSize. The file will be transferred in one write. # Chunked cannot be used in this case. $webRequest.AllowWriteStreamBuffering = $true; $webRequest.SendChunked = $false; $webRequest.ContentLength = $FileSize; } return $webRequest; } function Get-BufferSize { <# .SYNOPSIS Returns a buffer size that is 1% of ByteLength, rounded in whole MB or at least AtLeast size. .DESCRIPTION Returns a buffer size that is 1% of ByteLength, rounded to whole MB or if 1% is smaller than AtLeast, then AtLeast size is returned which is 1MB by default. .PARAMETER ByteLength Length of the bytes for which to calculate a valid buffer size. .PARAMETER AtLeast The minimum required buffer size, default 1MB. .EXAMPLE Get-BufferSize 4283304773 Returns 42991616 which is 41MB. .EXAMPLE Get-BufferSize 4283304 Returns 1048576 which is 1MB. .EXAMPLE Get-BufferSize 4283304 5MB Returns 5242880 which is 5MB. #> param( [Parameter(Mandatory=$true)] [long]$ByteLength, [long]$AtLeast = 1MB ) [long]$size = $ByteLength / 100; if ($size -lt $AtLeast) { $size = $AtLeast; } else { $size = [Math]::Round($size / 1MB) * 1MB; } return $size; } function Get-UseChunkedUpload { param( [Parameter(Mandatory=$true)] [long]$FileSize, [Parameter(Mandatory=$true)] [long]$BufferSize ) return $FileSize -gt $BufferSize; } function Configure-Tls { param( [Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Url ) [System.Uri]$uri = $Url; if ($uri.Scheme -eq "https") { Write-Verbose "Using TLS 1.2"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12; } } function Send-PreAuthenticate { param( [Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Url, [Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()] [pscredential]$Credential ) $response = $null; try { [System.Uri]$uri = $Url; $repositoryAuthority = (($uri.GetLeftPart([System.UriPartial]::Authority)).TrimEnd('/') + '/'); Write-Verbose "Send-PreAuthenticate - Sending HEAD to $repositoryAuthority"; $wr = [System.Net.WebRequest]::Create($repositoryAuthority); $wr.Method = "HEAD"; $wr.PreAuthenticate = $true; $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password)); $wr.Headers["Authorization"] = "Basic $auth" $response = $wr.GetResponse(); } finally { if ($response) { $response.Close(); $response.Dispose(); $response = $null; } } } function Upload-File { <# .SYNOPSIS Uploads a file to the Nexus repository. .DESCRIPTION Uploads a file to the Nexus repository. If the file was uploaded successfully, the url via which the resource can be downloaded is returned. .PARAMETER Url The Url where the resource should be created. Please note that underscores and dots should be encoded, otherwise the Nexus repository does not accept the upload. .PARAMETER File The file that should be uploaded. .PARAMETER Credential Credential used for authentication at the Nexus repository. .EXAMPLE Upload-File -Url https://nexusrepo.domain.com/repository/repo-name/myfolder/myfile%2Eexe -File (Get-ChildItem .\myfile.exe) -Credential (Get-Credential) .OUTPUTS If the file was uploaded successfully, the url via which the resource can be downloaded. #> param( [Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Url, [Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()] [System.IO.FileInfo]$File, [Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()] [pscredential]$Credential ) Write-Verbose "Upload-File Url:$Url" Configure-Tls -Url $Url; $fileSizeBytes = $File.Length; #$bufSize = Get-BufferSize $fileSizeBytes; $bufSize = 4 * 1MB; Write-Verbose ("FileSize is {0} bytes ({1:N0}MB). BufferSize is {2} bytes ({3:N0}MB)" -f $fileSizeBytes,($fileSizeBytes/1MB),$bufSize,($bufSize/1MB)); if (Get-UseChunkedUpload -FileSize $fileSizeBytes -BufferSize $bufSize) { Write-Verbose "Using chunked upload. Send pre-auth first."; Send-PreAuthenticate -Url $Url -Credential $Credential; } $progressActivityMessage = ("Sending file {0} - {1} bytes" -f $File.Name, $File.Length); $webRequest = New-HttpWebRequest -Url $Url -Credential $Credential -FileSize $fileSizeBytes -BufferSize $bufSize; $chunk = New-Object byte[] $bufSize; $bytesWritten = 0; $fileStream = [System.IO.File]::OpenRead($File.FullName); $requestStream = $WebRequest.GetRequestStream(); try { while($bytesRead = $fileStream.Read($chunk,0,$bufSize)) { $requestStream.Write($chunk, 0, $bytesRead); $requestStream.Flush(); $bytesWritten += $bytesRead; $progressStatusMessage = ("Sent {0} bytes - {1:N0}MB" -f $bytesWritten, ($bytesWritten / 1MB)); Write-Progress -Activity $progressActivityMessage -Status $progressStatusMessage -PercentComplete ($bytesWritten/$fileSizeBytes*100); } } catch { throw; } finally { if ($fileStream) { $fileStream.Close(); } if ($requestStream) { $requestStream.Close(); $requestStream.Dispose(); $requestStream = $null; } Write-Progress -Activity $progressActivityMessage -Completed; } # Read the response. $response = $null; try { $response = $webRequest.GetResponse(); Write-Verbose ("{0} responded with {1} at {2}" -f $response.Server,$response.StatusCode,$response.ResponseUri); return $response.ResponseUri; } catch { if ($_.Exception.InnerException -and ($_.Exception.InnerException -like "*bad request*")) { throw ("ERROR: " + $_.Exception.InnerException.Message + " Possibly the file already exists or the content type of the file does not match the file extension. In that case, disable MIME type validation on the server.") } throw; } finally { if ($response) { $response.Close(); $response.Dispose(); $response = $null; } if ($webRequest) { $webRequest = $null; } } } function Download-File { param( [Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$Url, [Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()] [string]$FileName ) $SDXDownloadType = @" using System.IO; using System.Net; public class SDXDownload { static public void DownloadFile(string Uri, string Filename) { HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(Uri); webRequest.Method = "GET"; using (HttpWebResponse myHttpWebResponse = (HttpWebResponse)webRequest.GetResponse()) using (Stream fileStream = File.OpenWrite(Filename)) using (Stream streamResponse = myHttpWebResponse.GetResponseStream()) { int bufSize = 64 * 1024; byte[] readBuff = new byte[bufSize]; int bytesRead = streamResponse.Read(readBuff, 0, bufSize); while (bytesRead > 0) { fileStream.Write(readBuff, 0, bytesRead); bytesRead = streamResponse.Read(readBuff, 0, 256); } } } } "@ Configure-Tls -Url $Url; Add-Type -TypeDefinition $SDXDownloadType; [SDXDownload]::DownloadFile($Url, $FileName); }