Upload XML file via FTP

Question

I have a list of channels in the database that I use to download an XML file from an FTP server, and then parse it. The script is inserted into the jar file, which is run daily using the Windows task manager. Sometimes a request occurs when capturing a specific XML file. Until now, this has happened about 3 times in 2 weeks without the real picture that I see.

When this goes bad, I go to the computer on which it runs, I see that the command window is open, and it stopped before the xml was fully loaded. If I close the command window and run the task manually, everything will work fine.

The code I use to download the xml file is:

private void loadFTPFile(String host, String username, String password, String filename, String localFilename){ System.out.println(localFilename); FTPClient client = new FTPClient(); FileOutputStream fos = null; try { client.connect(host); client.login(username, password); String localFilenameOutput = createFile(assetsPath + localFilename); fos = new FileOutputStream(localFilenameOutput); client.retrieveFile(filename, fos); } catch (IOException e) { e.printStackTrace(); } finally { try { if (fos != null) fos.close(); client.disconnect(); } catch (IOException e) { e.printStackTrace(); } } } 

This function is called in a loop, and when it fails, everything stops, and the script does not go to the next channel.

I'm not sure what is happening, possibly a loss of connection, but I think the / catch attempt will catch if that happens. I'm not sure if the timeout will use a trick or threads (but I have never worked with threads)

Can someone point me in the right direction on why this is happening and what I can do to fix the problem.

Answer 1

UPDATE - timeout setting for data connection

Since the last file was only partially downloaded, and given the source of FTPClient.retrieveFile() , I think this might be a problem on (something that makes it hang or even die - who knows). Obviously, you can’t repair the server or even know what is happening there, in any case, I suggest adding a timeout with setDataTimeout(int) and possibly catching SocketTimeoutException separately to register elsewhere and possibly be sent to the FTP server administrators (together with information about the time when this happened) so that they can combine the logs and see what the problem is.

OLD ANSWER

I did not notice that you are connecting and logging in for each file, so the following is just an optimization so as not to close the control connection and successfully log out, but it should not solve the problem.

You can run the JVM in debug mode and attach the debugger when it freezes, in any case, in accordance with this answer, and this thread may be a timeout problem on network equipment devices (routers). From FTPClient Javadoc

During file transfer, the data connection is busy, but the connection is idle. FTP servers know that the control connection is in use, therefore it will not close it due to insufficient activity, but it is much more difficult for network routers to know that the control and data connections are connected to each other. Some routers may handle the control of the connection in standby mode and disconnect it if the data transfer connection takes longer than the allowable idle time for the router.

One solution to this is to send a secure command (i.e., NOOP) over the control connection with the reset router idle timer. This is activated as follows:

 ftpClient.setControlKeepAliveTimeout(300); // set timeout to 5 minutes 

Answer 2

Do you check the return status of any of the calls or is it a code?

There is a call to completePendingCommand (), which should be used randomly. It may be something to learn.

In addition, you will not see an IO exception, I believe that it is repackaged as CopyStreamException

You might also want to change the return value to a boolean, because you are throwing exceptions, at least the call loop will know if the transition has occurred or not.

 private boolean loadFTPFile(String host, String username, String password, String filename, String localFilename){ System.out.println(localFilename); FTPClient client = new FTPClient(); FileOutputStream fos = null; try { client.connect(host); int reply = client.getReplyCode(); if (!FTPReply.isPositiveCompletion(reply)){ client.disconnect(); System.err.println("FTP server refused connection."); return false; } if (!client.login(username, password)){ client.logout(); return false; } String localFilenameOutput = createFile(assetsPath + localFilename); fos = new FileOutputStream(localFilenameOutput); boolean result = client.retrieveFile(filename, fos); SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS"); if (result){ System.out.println("\tFile Transfer Completed Successfully at: " + sdf.format(Calendar.getInstance().getTime())); // ftp.completePendingCommand(); } else { System.out.println("\tFile Transfer Failed at: " + sdf.format(Calendar.getInstance().getTime())); } return result; }catch (CopyStreamException cse){ System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.getInstance().getTime())); System.err.println("Error Occurred Retrieving File from Remote System, aborting...\n"); cse.printStackTrace(System.err); System.err.println("\n\nIOException Stack Trace that Caused the Error:\n"); cse.getIOException().printStackTrace(System.err); return false; }catch (Exception e){ System.err.println("\tFile Transfer Failed at: " + sdf.format(Calendar.getInstance().getTime())); System.out.println("Error Occurred Retrieving File from Remote System, aborting..."); e.printStackTrace(System.err); return false; } finally { try { if (fos != null) fos.close(); client.disconnect(); } catch (IOException e) { e.printStackTrace(); } } } 

Answer 3

This is not a thread issue. Most likely, this is caused by something in the loop, because this code looks as if it should be cleaned just fine. However, for testing, you probably want to add

 catch (Exception e) { e.printStackTrace(); } 

after an IOException catch clause. It is possible that another exception is raised.

Another thing is if you extract the results from the database result one at a time and do FTP access, this can be a problem. If the results are not returned immediately by a JDBC call, this may also end. Not all database queries actually return the entire result set to the client right away.