Output cat pipe to cURL to load a list of files

Question

I have a list url in a file called urls.txt . Each line contains 1 URL. I want to download all files at once using cURL. I can't seem to get the right single-line snapshot.

I tried:

 $ cat urls.txt | xargs -0 curl -O 

But this gives me only the last file in the list.

Answer 1

This works for me:

 $ xargs -n 1 curl -O < urls.txt 

I'm on FreeBSD. Your xarg may work differently.

Answer 2

A very simple solution would be the following: If you have a file 'file.txt', for example

 url="http://www.google.de" url="http://www.yahoo.de" url="http://www.bing.de" 

Then you can use curl and just do

 curl -K file.txt 

And curl will call all Urls contained in file.txt!

So, if you have control over the file input format, perhaps this is the easiest solution for you!

Answer 3

Or you could just do this:

 cat urls.txt | xargs curl -O 

You need to use the -I option if you want to insert cat output in the middle of the command.

Answer 4

Here's how I do it on a Mac (OSX), but it should work equally well on other systems:

What you need is a text file containing your links for curl

So:

  http://www.site1.com/subdirectory/file1-[01-15].jpg http://www.site1.com/subdirectory/file2-[01-15].jpg . . http://www.site1.com/subdirectory/file3287-[01-15].jpg 

In this hypothetical case, the text file has 3287 lines, and each line encodes 15 images.

Let's say we save these links in the text file testcurl.txt at the top level (/) of our hard drive.

Now we need to go into the terminal and enter the following command in the bash shell:

  for i in "`cat /testcurl.txt`" ; do curl -O "$i" ; done 

Make sure you use back ticks (`) Also make sure that the (-O) flag is capital O and NOT zero

with the -O flag, the original file name will be written

Happy download!

Answer 5

xargs -P 10 from GNU downloads files in parallel up to 10 threads:

 xargs -P 10 -n 1 curl -O < urls.txt 

This will speed up 10x downloads if your maximum download speed, if not reached, and if the server does not throttle IP addresses, which is the most common scenario.

Just do not set the -P level too high or your RAM may be overloaded.

GNU parallel can achieve similar results.

The disadvantage of these methods is that they do not use the same connection for all files, which makes curl if you pass it several URLs at once:

 curl -O out1.txt http://exmple.com/1 -O out2.txt http://exmple.com/2 

as stated at https://serverfault.com/questions/199434/how-do-i-make-curl-use-keepalive-from-the-command-line

Maybe combining both methods will give the best results? But I believe that parallelization is more important than keeping in touch.

See also: Parallel Download Using Curl Command-Line Utility

Answer 6

As others rightly mentioned:

 -cat urls.txt | xargs -0 curl -O +cat urls.txt | xargs -n1 curl -O 

However, this paradigm is a very bad idea, especially if all your URLs come from the same server - you will not only create another curl instance, but also will establish a new TCP connection for each request that is extremely inefficient and especially with the ubiquitous https.

Use instead:

 -cat urls.txt | xargs -n1 curl -O +cat urls.txt | wget -i/dev/fd/0 

Or, even simpler:

 -cat urls.txt | wget -i/dev/fd/0 +wget -i/dev/fd/0 < urls.txt 

The easiest way:

 -wget -i/dev/fd/0 < urls.txt +wget -iurls.txt