How to skip already existing files when loading with curl?

Question

I want curl download a link, but I want it to skip files that already exist. Right now, the line of code that I have will continue to overwrite it without any changes:

 curl '$url' -o /home/$outputfile &>/dev/null & 

How can this be achieved?

Answer 1

You can simply put your call to curl inside the if block:

 if ! [ -f /home/$outputfile ]; then curl -o /home/$outputfile "url" fi 

Also note that in your example, you have $url inside single quotes that won't do what you want. For comparison:

 echo '$HOME' 

To:

 echo "$HOME" 

In addition, curl has the --silent , which can be useful in scripts.

Answer 2

Use wget with --no-clobber :

-nc , --no-clobber : skip downloads that will be loaded into existing files.

Example:

 wget -nc -q -O "/home/$outputfile" "$url" 

Answer 3

You can use the curl function -C - . This option is used to resume a downloaded download, but the download will be skipped if the file is already completed. Note that the -C argument is a dash. The disadvantage may be that the curl connects to the remote server for a short while to set the file size.

Answer 4

curl may support file skips when used with -O and -J , but its behavior is incompatible.

-J ( --remote-header-name ) basically tells the -O ( --remote-name ) option to use the specified Content-Disposition server name, rather than retrieving the file name from the URL. Thus, curl does not know which file name will be returned by the server, so it can ignore an existing file for security.

^{Source: Re: -J "Failure to overwrite ..."}

For example:

 $ curl -LJO -H 'Accept: application/octet-stream' 'https://api.github.com/repos/x/y/releases/assets/12345 Warning: Refusing to overwrite my_file.bin: File Warning: exists curl: (23) Failed writing body (0 != 16384) 

However, as already mentioned, its behavior is unpredictable and does not work for all files.