The .lastModified () file is painfully slow!

Question

I make a recursive copy of files and love xcopy /D I only want to copy files with newer files (I cannot use xcopy directly, since I need to change some files during the copy process).

In java, I use lastModified() to check if the target file is older than the original file and it is very slow.

• Can I speed up the process (possibly using JNI?)?
• Are there other copy scripts that can do the job better (copy new files + regexp, change some text files)?

Copying files in any case is not an option, since it will take longer than checking the last modified date (copying over the network).

Answer 1

You need to determine why it is so slow.

When you run progrma, what is the use of your process in the CPU. If it is more than 50% of users, then you should be able to choose your program, if it is less than 20%, not much can be done.

Usually this method is slow because the file being scanned is located on disk, not in memory. If so, you need to speed up access to your disk or get a faster disk. For example, an SSD can be 10-100 times faster.

Bulk request may be required. You can do this using multiple threads to check the lastModified date. For example, they have a fixed-size thread pool and add a task for each file. The thread pool size determines the number of polled files at one time.

This allows the OS to reorder requests according to the layout on the disk. Note. This is theoretically, but you need to check if it speeds up on your OS / hardware, as this can slow down.;)

Answer 2

Unfortunately, the lastModified Java processing method is slow (basically it asks for a basic file system for each file as information is requested, there is no bulk loading of this data into listFiles or similar).

You could potentially invoke a more efficient native program to do it in bulk, but any such solution would be closely related to the platform you are deploying to.

Answer 3

I assume that you are doing this over the network, otherwise the copy will make little sense. Network directory operations are slow, unsuccessful. You can always simply copy a file under a threshold of a certain size, regardless of the fact that the general operation takes less time.

I disagree with Chris here: there is nothing strikingly inefficient in how Java does it, and in any case, it really needs to be done so because you need the last value.

Answer 4

Have you verified that you have tested this method?

Answer 5

So, I came across this on network drives. Painful. I had a directory with 17000 + files. It took less than 2 seconds on the local drive to check the last modified date. On a network drive, it took 58 seconds !!! Of course, my application is an interactive application, so I had some complaints.

After some research, I decided it would be possible to implement some JNI code to make Windows Kernel32 findfirstfile / findnextfile / findclose to greatly improve the process, but then I had a 32-bit and 64-bit version, etc. and then lose cross platform capabilities.

Although a bit of a nasty hack here is what I did. My application works mainly on Windows, but I did not want to limit it, so I did the following. Check if I work on windows. If yes, then see if I use a local hard drive. If not, we are going to make a hacker method.

I kept everything case insensitive. This is probably not a great idea for other operating systems that may have a directory with both "ABC" and "abc". If you need to do this, you can decide by creating a new file ("ABC") and a new file ("abc"), and then use the equals method to compare them. On case-insensitive file systems such as windows, it will return true, but on Unix systems it will return false.

Although this may be a bit hacky, the time spent on it lasted from 58 seconds to 1.6 seconds on a network drive, so I can live with a hack.

  boolean useJaveDefaultMethod = true; if(System.getProperty("os.name").startsWith("Windows")) { File f2 = f.getParentFile(); while(true) { if(f2.getParentFile() == null) { String s = FileSystemView.getFileSystemView().getSystemTypeDescription(f2); if(FileSystemView.getFileSystemView().isDrive(f2) && "Local Disk".equalsIgnoreCase(s)) { useJaveDefaultMethod = true; } else { useJaveDefaultMethod = false; } break; } f2 = f2.getParentFile(); } } if(!useJaveDefaultMethod) { try { ProcessBuilder pb = new ProcessBuilder("cmd.exe", "/C", "dir " + f.getParent()); pb.redirectErrorStream(true); Process process = pb.start(); InputStreamReader isr = new InputStreamReader(process.getInputStream()); BufferedReader br = new BufferedReader(isr); String line; DateFormat df = new SimpleDateFormat("dd-MMM-yy hh:mm a"); while((line = br.readLine()) != null) { try { Date filedate = df.parse(line); String filename = line.substring(38); dirCache.put(filename.toLowerCase(), filedate.getTime()); } catch(Exception ex) { } } process.waitFor(); Long filetime = dirCache.get(f.getName().toLowerCase()); if(filetime != null) return filetime; } catch(Exception Exception) { } } // this is SO SLOW on a networked drive! long lastModifiedDate = f.lastModified(); dirCache.put(f.getName().toLowerCase(), lastModifiedDate); return lastModifiedDate;