How can I atomize the creation of a locked file in Linux?

Question

Scenario: I have many processes that need to be downloaded over the network. If the file is already loaded, I want to cache it on disk. If another process is downloading the file, lock it until the download is complete.

I am trying to find the easiest way to do this. The obvious way is:

create file w/ an exclusive lock active on it only if it doesn't exist (O_CREAT | O_EXCL) if file exists already: open file and acquire exclusive lock else: download to newly created file release lock 

This system fulfills the above goals with (seemingly) no race conditions

Unfortunately, I could not find the documentation on how to use open (), etc. to create a file that is locked on Linux. If I divided the creation step into:

 open w/ O_CREAT | O_EXCL flock 

there is now a race condition between creation and locking (the non-creating process gets the lock before the creator does).

I understand that I can use an external lock file for each file (for example, file_name + .lock) that I acquired before trying to create the file name, but that seems .. inelegant (and now I need to worry about how the files are, which the suffix .lock actually has!)

Is there anyway to atomize and block it (as suggested by Windows), or is it an external file locking method to a large extent, which is standard / mandatory?

Answer 1

The race exists anyway. If a file may or may not exist, you must verify its existence before attempting to block it. But if the file is your mutex, then you cannot do this, and the space between "if file exists is" (false) and "upload to a new file" is unlimited. Another process may arise and create the file and start the download before the download starts, and you can hide it.

Basically, fcntl locks are not used here, use the existence of the file itself. open() with O_CREAT and O_EXCL will fail if the file already exists, telling you that someone else got there first.

Answer 2

Why aren't you using the lockfile utility?

Examples

Suppose you want to make sure that access to the file is “important” serialized, that is, no more than one program or shell script should be allowed to access it. For simplicity, suppose this is a shell script. In this case, you can solve it as follows:

 ... lockfile important.lock ... access_"important"_to_your_hearts_content ... rm -f important.lock ...