How to optimize loading with Delphi 2010?

Question

My is not yet released . Delphi 2010 application allows users to upload their files to my servers. Right now I am using HTTPS POST to send files, the algorithm (simplified) is basically:

• Split file into slices (256 KB each)
• For each fragment, send it to the server

T. for a 1 MB file:

--> Get Slice #1 (256KB) --> Upload Slice #1 using TidHTTP.Post() --> Get Slice #2 (256KB) --> Upload Slice #2 using TidHTTP.Post() --> Get Slice #3 (256KB) --> Upload Slice #3 using TidHTTP.Post() --> Get Slice #4 (256KB) --> Upload Slice #4 using TidHTTP.Post() 

I am using Indy 10 . I (ab) used my profiler again and again, and there’s nothing to optimize except changing the boot procedure itself.

I also use multithreading , and although I tried to optimize my code, my tests still tell me what I can do better (there are other well-optimized programs that achieve significantly better time ... almost twice as fast as my program downloads!)

I know this is not my server error ... here are some ideas that I still need to explore:

• I tried to group slices in one POST, of course, this led to an increase in productivity (20-35%), but the renewal ability is now reduced.

• I also thought about using SFTP / SSH, but I'm not sure if it is fast.

• Use web sockets to implement renewable downloads (like this component ), I'm also not sure about the speed.

Now my question is: is there anything I can do to speed up the download? I am open for any offer that I can implement, including command line tools (if the license allows me to send it with my application), provided that:

• Supported by renewable download.
• Fast!
• Reasonable memory usage
• Protect and allow user / user authentication

Also, due to serious security issues, FTP is not what I want to implement.

Thank you so much!

Answer 1

I would suggest making a single TIdHTTP.Post() for the whole file without traversing it. You can use the TIdHTTP.OnWork... events to keep track of how many bytes were sent to the server, so you know where to resume work if necessary. When resuming, you can use the TIdHTTP.Request.CustomHeaders property to include a custom header that tells the server from which you are resuming, so it can roll back its previous file to the specified offset before accepting new data.