Java Download a massive file giving connection / reset termination on the Internet at the URL

Question

I am creating a swing application to upload multiple files over the Internet and save to a windows file. I used SwingWroker, which internally uses the ExecutorService, which internally queues them and downloads 10 at a time, but for some reason, after downloading, say, 2 - 3 MB of the file, it stops and moves on to the next file to be downloaded. They are loaded into a package of 10 since SwingWorker fixed it in the number of threads for the executing service.

I need to write these files to a shared Windows file, and for this I use nio.FileChannels. There are files in the range from 50 to 60 each weighing about 300 MB - 500 MB. Links to files are placed on the web page where I get login using the credentials on the login page (with the mail request) via the Internet, before I specify CookieHandler.setDefault(new CookieManager()) at the beginning, and therefore it leads like a browser for me

Another observation is that when I upload them locally (and not to the Windows server share), they work fine.

This is the code I'm using.

 import java.io.File; import java.io.FileOutputStream; import java.net.URL; import java.net.URLConnection; import java.nio.channels.Channels; import java.nio.channels.FileChannel; import java.nio.channels.ReadableByteChannel; import javax.swing.SwingWorker; public class DownloadProcess extends SwingWorker<Boolean, String> { private String urlPath, filePath; public DownloadProcess(String urlPath, String filePath){ this.urlPath = urlPath; this.filePath = filePath; } @Override protected Boolean doInBackground() { boolean taskState = true; URLConnection httpConn = null; ReadableByteChannel readableByteChannel = null; FileOutputStream fileOutputStream = null; FileChannel fileOutputChannel = null; try{ //String filePath = "\\\\fileshare.server\\xyz.txt"; //String urlPath = "http://example.com/anyBigFile.1GB.docx"; File localFile = new File(filePath);//File share boolean itsThere = localFile!=null && localFile.exists(); long done = itsThere ? localFile.length() : 0; URL url = new URL(urlPath); httpConn = url.openConnection(); httpConn.setRequestProperty("Connection", "keep-alive"); if(itsThere) { httpConn.setRequestProperty("Range","bytes="+done+"-"); } readableByteChannel = Channels.newChannel(httpConn.getInputStream()); fileOutputStream = itsThere ? new FileOutputStream(filePath) : new FileOutputStream(filePath,true); fileOutputChannel = fileOutputStream.getChannel(); for (long position = done, size = httpConn.getContentLength(); position < size && !isCancelled(); ) { position += fileOutputChannel.transferFrom(readableByteChannel, position, 1 << 16); } //done }catch(Exception e){ taskState = false; e.printStackTrace(); }finally{ //close streams conns etc } return taskState; } } 

This is the error stack trace that I get after 5-10 minutes of loading

 /* javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset at sun.security.ssl.SSLSocketImpl.checkEOF(Unknown Source) at sun.security.ssl.AppInputStream.read(Unknown Source) at java.io.BufferedInputStream.read1(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at sun.net.www.MeteredStream.read(Unknown Source) at java.io.FilterInputStream.read(Unknown Source) at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source) at java.nio.channels.Channels$ReadableByteChannelImpl.read(Unknown Source) at com.objects.DownloadByteChannel.read(DownloadByteChannel.java:117) at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(Unknown Source) at sun.nio.ch.FileChannelImpl.transferFrom(Unknown Source) at com.core.DownloadTask.doInBackground(DownloadTask.java:154) at com.core.DownloadTask.doInBackground(DownloadTask.java:59) at com.util.ZSwingWorker$1.call(ZSwingWorker.java:286) at java.util.concurrent.FutureTask.run(Unknown Source) at com.util.ZSwingWorker.run(ZSwingWorker.java:325) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset at sun.security.ssl.Alerts.getSSLException(Unknown Source) at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source) at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source) at sun.security.ssl.SSLSocketImpl.handleException(Unknown Source) at sun.security.ssl.SSLSocketImpl.handleException(Unknown Source) ... 18 more Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at sun.security.ssl.InputRecord.readFully(Unknown Source) at sun.security.ssl.InputRecord.read(Unknown Source) at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source) at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source) ... 18 more */ 

Using:

 public static void main(String[] args){ int counter = 1; for(String url: urls){ new DownloadProcess(url,"\\\\fileshare.server\\xyz"+(counter++)+".txt").execute(); } } 

Answer 1

You will have to change the timeout for connecting to the server. I took a few links along the way if they have any meaning:

Change Session Security Settings

Extending Salesforce Session Timeout

Hope this helps, good luck and let me know :)

Answer 2

A Reset connection means that the remote side closes the connection to the TCP RST packet (reset). You need to find out what the remote side is not like and fix it.

If the remote side is Apache, you are probably using the value KeepAliveTimeout. The default is 5 seconds. It looks like you are bumping into some kind of configured limit on the far side. When this happens, the server drops you using reset.