Wget with input file and output document

Question

I have a list of URLs that I would like to use in wget using --input-file.

However, I cannot decide how to control the value of -output-document at the same time, which is simple if you issue the commands one by one. I would like to save each document as MD5 its URL.

cat url-list.txt | xargs -P 4 wget 

And xargs exists because I also want to use max-procs functions for parallel downloads.

Answer 1

how about using a loop?

 while read -r line do md5=$(echo "$line"|md5sum) wget ... $line ... --output-document $md5 ...... done < url-list.txt 

Answer 2

Do not use cat . You can read xargs from a file. On the man page:

  --arg-file = file
        -a file
               Read items from file instead of standard input.  If you use this
               option, stdin remains unchanged when commands are run.  Other‐
               wise, stdin is redirected from / dev / null.

Answer 3

In your question, you use -P 4, which suggests that your solution be executed in parallel. GNU Parallel http://www.gnu.org/software/parallel/ can help you:

 cat url-list.txt | parallel 'wget {} --output-document "`echo {}|md5sum`"' 

Answer 4

You can do it like this:

cat url-list.txt | when reading url; do wget $ url -O $ (echo "$ url" | md5); done

luck