I am trying to create a collection of FTP web requests to download a collection of files.
Worked correctly on this in a single thread, but now I'm trying to make multiple threads, but I get a timeout exception. I think I'm missing something quite simple, but it doesn't seem to work.
Here is the code:
internal static void DownloadLogFiles(IEnumerable<string> ftpFileNames, string localLogsFolder) { BotFinder.DeleteAllFilesFromDirectory(localLogsFolder); var ftpWebRequests = new Collection<FtpWebRequest>(); // Create web request for each log filename foreach (var ftpWebRequest in ftpFileNames.Select(filename => (FtpWebRequest) WebRequest.Create(filename))) { ftpWebRequest.Credentials = new NetworkCredential(BotFinderSettings.FtpUserId, BotFinderSettings.FtpPassword); ftpWebRequest.KeepAlive = false; ftpWebRequest.UseBinary = true; ftpWebRequest.CachePolicy = NoCachePolicy; ftpWebRequest.Method = WebRequestMethods.Ftp.DownloadFile; ftpWebRequests.Add(ftpWebRequest); } var threadDoneEvents = new ManualResetEvent[ftpWebRequests.Count]; for (var x = 0; x < ftpWebRequests.Count; x++) { var ftpWebRequest = ftpWebRequests[x]; threadDoneEvents[x] = new ManualResetEvent(false); var threadedFtpDownloader = new ThreadedFtpDownloader(ftpWebRequest, threadDoneEvents[x]); ThreadPool.QueueUserWorkItem(threadedFtpDownloader.PerformFtpRequest, localLogsFolder); } WaitHandle.WaitAll(threadDoneEvents); } class ThreadedFtpDownloader { private ManualResetEvent threadDoneEvent; private readonly FtpWebRequest ftpWebRequest; /// <summary> /// /// </summary> public ThreadedFtpDownloader(FtpWebRequest ftpWebRequest, ManualResetEvent threadDoneEvent) { this.threadDoneEvent = threadDoneEvent; this.ftpWebRequest = ftpWebRequest; } /// <summary> /// /// </summary> /// <param name="localLogsFolder"> /// /// </param> internal void PerformFtpRequest(object localLogsFolder) { try { // TIMEOUT IS HAPPENING ON LINE BELOW using (var response = ftpWebRequest.GetResponse()) { using (var responseStream = response.GetResponseStream()) { const int length = 1024*10; var buffer = new Byte[length]; var bytesRead = responseStream.Read(buffer, 0, length); var logFileToCreate = string.Format("{0}{1}{2}", localLogsFolder, ftpWebRequest.RequestUri.Segments[3].Replace("/", "-"), ftpWebRequest.RequestUri.Segments[4]); using (var writeStream = new FileStream(logFileToCreate, FileMode.OpenOrCreate)) { while (bytesRead > 0) { writeStream.Write(buffer, 0, bytesRead); bytesRead = responseStream.Read(buffer, 0, length); } } } } threadDoneEvent.Set(); } catch (Exception exception) { BotFinder.HandleExceptionAndExit(exception); } } }
It seems to load the first two files (using the two streams that I accept), but then it seems like a timeout occurs when these full and applications try to move to the next file.
I can confirm that FTPWebRequest, which is a timeout, is valid and the file exists, I think I can have an open connection or something like that.
I was going to leave a comment, but probably easier to read in response:
First, if I set the ftpRequest.Timout property to Timeout.Infinite, the timeout problem disappears, however, an infinite timeout is probably not the best way. Therefore, I would prefer to solve it in a different way ...
Debugging the code, I see that when it gets to:
ThreadPool.QueueUserWorkItem(threadedFtpDownloader.PerformFtpRequest, localLogsFolder);
He introduces the PerformFtpRequest method for each FTP web request and calls ftpWebRequest.GetResponse (), but then only moves forward for the first two requests. The remaining requests remain active, but do not go any further until the first two are over. Thus, this basically means that they remain open, waiting for other requests to complete before running.
I think that the solution to this problem would be to either allow all requests to be executed immediately (the ConnectionLimit property does not work here), or to prevent the GetResponse call from being executed until it is ready to use the response.
Any good ideas on the best way to solve this? At the moment, all I can imagine is hacking solutions that I would like to avoid :)
Thanks!