Running Invoke-WebRequest Parallel Jobs in PowerShell v3

Question

Running simultaneous background jobs in PowerShell is pretty simple, but I can't get it to work with the new (in v3) Invoke-WebRequest cmdlet.

I have several thousand files that I download scripted through PowerShell. It works fine, but it takes several days in the series:

for($f=0;$f -lt $urlList.Count;$f++) { $remote = $urlList[$f] + $fileList[$f] $local = 'C:\folder\' + $fileList[$f] Invoke-WebRequest $remote -Method Get -OutFile $local -UserAgent FireFox } 

I made several attempts to use the "AsJob" method, but either they are crossed out or finished, but local files will not be saved. Here is an example of the latter:

 for($f=0;$f -lt $urlList.Count;$f++) { $remote = $urlList[$f] + $fileList[$f] $local = 'C:\folder\' + $fileList[$f] $command = "Invoke-WebRequest $remote -Method Get -OutFile $local -UserAgent FireFox" Start-Job {Invoke-Expression -Command $command} } Get-Job|Wait-Job 

Output Examples:

 Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 339 Job339 BackgroundJob Running True localhost Invoke-Expression -Com... 341 Job341 BackgroundJob Running True localhost Invoke-Expression -Com... 343 Job343 BackgroundJob Running True localhost Invoke-Expression -Com... 339 Job355 BackgroundJob Completed True localhost Invoke-Expression -Com... 341 Job357 BackgroundJob Completed True localhost Invoke-Expression -Com... 343 Job359 BackgroundJob Completed True localhost Invoke-Expression -Com... 

It’s strange that the tasks above are about as fast as required to download the linked file ... so it seems that the data is going somewhere - just not in my save location. I assume this happens in memory, but is reset without saving. I tried to add and remove the parameter "PassThru", but get the same results anyway. I also tried to connect it to Out-File, still without any joy. I don’t know why this particular cmdlet is such a fagot.

Answer 1

You need to pass the parameters in such a way that they actually fall into the task. In V3, you can use the $ using: syntax to do this:

 for($f=0;$f -lt $urlList.Count;$f++) { $remote = $urlList[$f] + $fileList[$f] $local = 'C:\folder\' + $fileList[$f] Start-Job {Invoke-WebRequest $using:remote -Method Get -OutFile $using:local -UserAgent FireFox} } Get-Job|Wait-Job 

By the way, you do not need to use Invoke-Expression. Just enter the code that you want to run in the task inside the script block, that is, a set of curly braces {...} .