Java: how to handle two processes trying to change the same file

Question

Possible duplicate:
How to lock a file using java (if possible)

I have two processes that invoke two Java programs that modify the same text file. I noticed that the contents of the text file are missing data. I suspect that when one Java program receives a stream of writes to a text file, I think that it blocks another java program from modifying it (for example, when you have a file open, you cannot delete this file). Is there any way around this other than the database? (not to say that the db solution is not clean or elegant, we just wrote a lot of codes to manipulate this text file)

EDIT

It turns out that I made a mistake aimed at the problem. The reason the data in my text file is missing is because

ProcessA : Add Data Rows to a Text File

ProcessB : First load all the lines of the text field in the List . Then it processes the list of this list. At the end of ProcessB write a list, replacing the contents of the text file.

This work is great in a consistent process. But when sharing, if ProcessA adding data to a file, during ProcessB manipulating the List , then when ProcessB writes List , regardless of what only ProcessA just adds, it will be overridden, Therefore, my initial thought was before ProcessB to write List back, synchronize data between text file and List . Therefore, when I write List back, it will contain everything. so here is my effort

 public void synchronizeFile(){ try { File file = new File("path/to/file/that/both/A/and/B/write/to"); FileChannel channel = new RandomAccessFile(file, "rw").getChannel(); FileLock lock = channel.lock(); //Lock the file. Block until release the lock List<PackageLog> tempList = readAllLogs(file); if(tempList.size() > logList.size()){ //data are in corrupted state. Synchronized them. for(PackageLog pl : tempList){ if(!pl.equals(lookUp(pl.getPackageLabel().getPackageId(), pl.getPackageLabel().getTransactionId()))){ logList.add(pl); } } } lock.release(); //Release the file channel.close(); } catch (IOException e) { logger.error("IOException: ", e); } } 

So logList is the current list that ProcessB wants to write. Therefore, before writing, I read the file and saved the data in tempList , if tempList and logList do not match, synchronize them. The problem is that at the moment, both ProcessA and ProcessB are currently accessing the file, so when I try to lock the file and read from it List<PackageLog> tempList = readAllLogs(file); , I either get an OverlappingFileLockException or java.io.IOException: The process cannot access the file because another process has locked a portion of the file . Please help me solve this problem :(

EDIT2 : I understand Lock

 public static void main(String[] args){ File file = new File("C:\\dev\\harry\\data.txt"); FileReader fileReader = null; BufferedReader bufferedReader = null; FileChannel channel = null; FileLock lock = null; try{ channel = new RandomAccessFile(file, "rw").getChannel(); lock = channel.lock(); fileReader = new FileReader(file); bufferedReader = new BufferedReader(fileReader); String data; while((data = bufferedReader.readLine()) != null){ System.out.println(data); } }catch(IOException e){ e.printStackTrace(); }finally{ try { lock.release(); channel.close(); if(bufferedReader != null) bufferedReader.close(); if(fileReader != null) fileReader.close(); } catch (IOException e) { e.printStackTrace(); } } } 

and I got this error IOException: The process cannot access the file because another process has locked a portion of the file

Answer 1

So, you can use the method that Vineet Reynolds offers in your comment.

If two processes are actually just separate threads in one application, then you can set a flag somewhere to indicate that the file is open.

If these are two separate applications / processes in general, the main file system should block the files. When you get an I / O error from the output stream, you can wrap a try / catch block around it and then configure the application to try again later or any desired behavior for your specific application.

Files are not intended to be written simultaneously by several applications. If you can describe why you want to write to a file from several processes simultaneously, there may be other solutions that could be suggested.

---

Updates after the latest changes: Good, so you need at least 3 files to do what you are talking about. You definitely cannot read and write data to a single file at the same time. Your three files:

• file that ProcessA dumps new / incoming data to
• the file that ProcessB is currently working on
• The final "output" file containing the output from ProcessB.

ProcessB cycle:

• Take any data in file # 2, process it and write the output to file # 3
• Delete file # 2
• Repeat

ProcessA cycle:

• Enter all new input into file # 1
• Periodically check if file # 2 exists
• When file # 2 is deleted by ProcessB, ProcessA must stop writing to file # 1, rename file # 1 to file # 2
• Run the new file # 1
• Repeat

Answer 2

If these are two separate applications trying to access the file. One of them will be through an IOException, because it cannot receive it. If this happens, add code to catch(IOException err){} to pause the current thread for a few milliseconds, and then recursively try to write again - until it gets access.

 public boolean writeFile() { try { //write to file here return true; } catch (IOException err) // Can't access { try { Thread.sleep(200); // Sleep a bit return writeFile(); // Try again } catch (InterruptedException err2) { return writeFile(); // Could not sleep, try again anyway } } } 

This will continue until you get a StackOverflow Exception , which means it's too deep; but the likelihood that this happens in this situation is very small - it will only happen if the file is stored for a long time by another application.

Hope this helps!

Answer 3

The code in the updated question most likely refers to process B and not to process A. I assume that it is.

Given that an OverlappingFileLockException is thrown, it seems that another thread in the same process is trying to lock the same file. This is not a conflict between A and B, but rather a conflict inside B if someone follows the API documentation using the lock () method and when the condition under which it throws an OverlappingFileLockException:

If a lock that overlaps the requested region is already supported by this Java virtual machine, or if another thread in this method is already blocked, it tries to block the overlapping region of the same file

The only solution to prevent this is to prevent any other thread in B from getting a lock in the same file or the same overlap area in the file.

Throwing an IOException has a slightly more interesting message. This probably confirms the above theory, but without looking at the entire source code, I cannot confirm anything. It is expected that the lock method will be locked until an exclusive lock is received. If it was purchased, then there should not be any problems in the letter to the file. Except for one condition. If the file has already been opened (and locked) by the same JVM in another thread using the File object (or, in other words, the second / other file descriptor), then an attempt to write to the first file descriptor will fail even if the lock was acquired (in after all, blocking does not block other threads).

An improved design should consist in creating a single thread in each process that receives an exceptional file lock (when using one File object or one file descriptor) for only a certain amount of time, performs what is required in the file, and then release the lock.

Answer 4

Think about it using the MapReduce mentality. Suppose each program writes output without reading another output. I would write two separate files, and then had the “reduce” step. Your contraction may be a simple chronologically ordered merger.

If, however, your programs require one-way output. You have a completely different problem, and you need to rethink how you break up the work.

Finally, if the outputs of the two programs are similar, but independent, and you write them to a single file so that the third program can read all of this, consider changing the third program to read both files.