Work with incremental server response in AJAX (in JavaScript)

Question

I have an AJAX application that refreshes a page based on server response. The command on which the AJAX server response is based takes a long time to generate a complete response, but sends partial information as soon as it is calculated. This partial response / partial information is sent in a “packet”, and the time and size of each packet is unpredictable. A CGI script (in Perl) that sends an output command to a web browser (upon AJAX request) has autorun enabled.

The server response is based on the output of an external command. While "time cmd> / dev / null" averages around 10.0 seconds, "time cmd | head> / dev / null" gives less than 0.1 seconds (for example, data). All data is the result of a single call to this external command.

The situation is as follows (ASCII-art drawing):

client | | server --------- --------- request -\ \ \ \ \-> /- response / . / . / /- . <-/ / . / . / /- [end] <-/ / / / <-/ 

I have a few questions about this issue.

Note: the server side runs as a CGI script in Perl, and I would prefer to see (also) a solution without using a JavaScript / framework library like jQuery.

• The output of the command used on the server side of the AJAX application is line-based. Each row group, starting with one particular row type and ending with another row type, consists of independent and immutable data. Should I just pass the response from the command as “text / plain” and process it in client-side JavaScript, or should I pre-process the data on the server and send entire pieces of data in JSON using the “application / json” mimetype?

• It may happen that a large chunk of data sent simultaneously by the server will soon be executed by another chunk of data. How to deal with the situation when the onreadystatechange handler onreadystatechange called before the previous call has completed its work? Should I use a global variable as a semaphore or pass a state variable as a parameter to a handler (well, use xhr.onreadystatechange = function() { handleRequest(xhr, state) } )?

• Should I use 'text / plain' or 'application / json', or perhaps 'multipart / x0mixed-replace' for this? Note. This application should work in (alomst) in any browser.

• How to work with a web browser (JavaScript mechanisms) that call onReadyStateChange only after receiving a full response (therefore, I do not see xhr.readyState == 3 , that is, a partial response more than once)? Well, besides using some JavaScript frameworks.

• How to deal with incomplete answers (which in this situation mean incomplete lines).

• Should I send an end-of-reply marker or rely on a counter to check if we have received all the data, or can I just expect xhr.readyState == 4 to be detected?

Even a partial answer would be helpful.

Answer 1

I think Comet is part of what you need for your solution. You can additionally (if I got this right) check the Bayeux protocol , which was implemented by the Dojo Foundation. All of this is still very new (although some may be possible with the first implementations of HTML5).

In addition, you may have to implement a survey approach. Another question: how much data can the JavaScript interpreter process on the client side. Do you have the ability to somehow "output" your data so that you do not have problems with processing requests that are still being processed, while another answer comes already?

Answer 2

I think the client side can be designed to process the data in pieces, send repeated AJAX requests until all the data is provided. This assumes that each piece can be sent on time (without timeout problems on the client side), even if the whole response was large; and this design is probably simpler than developing checks for certain partial response statuses, which may vary from browser to browser.

In terms of whether you are processing server-side or client-side processing, it depends, for example, how many concurrent clients you will need to serve, and whether you can use caching for any of the responses; in some scenarios where there are many clients, it is better to distribute part of the load on them (so far they can handle this, of course).