In a fairly broad and general way of describing things (i.e., not specific to .net or any other platform), services are programmed in one of two ways or a combination of both.
thread per connection: as you noticed, it uses a single scheduling object, such as a thread or a kernel process, or, possibly, a lighter thread or a joint routine implemented by the platform in user space, so that each process requires only one request processing , OK.
pseudo code example:
function run(connection) while(connection is open) frobnicate(connection) function main listener = new Listener(port) threads = new Collection<Thread>() while(running) connection = listener.accept() threads.add(new Thread(run, connection)) join(threads) exit()
Important features of the above example:
- this follows the flow onto the connection model, as you put it. Once the thread has completed work on the connection, it dies.
- If this program is to run in the real world, it will probably be okay before it has to process more than a few hundred threads. Most platforms begin this level. Some of them are specially designed to handle many threads, in tens of thousands of simultaneous connections.
- Although connections are handled in a way that is multi-threaded, in reality, incoming connections are only processed in the main thread, it is possible to be a DoS point if the malicious host tries to connect, but intentionally fails, slowly.
- When the program detects that it is time to stop working, the main thread joins another so that they do not die when they exit. again, this means that the program will not exit until all the old threads have finished processing the data, which may just never happen. An alternative would be to not use the connection, in which case other flows would be unceremoniously terminated.
non-blocking: one thread of execution manages several connections, and when any given connection is stalled, because it expects data to move across the network or will be read from disk, the process skips it and works on another connection.
pseudo code example:
function main connections = new Collection<Connection>() listener = new Listener() connections.append(listener) foreach connection in connections: if connection.ready(): if connection is listener: connections.add(connection.accept() ) else: if connection is open: nb_frobnicate(connection) else: connections.remove(connection) yield() if( not running ) exit()
Features of this fragment:
- In the multi-threaded version, each thread processed only one connection. if this connection is not ready for use, flow blocks and another thread can perform Work. if
frobnicate blocked in this implementation, it will be a disaster. Other connections may work even if one connection is not ready. To solve this problem, an alternative function is used, which allows you to use only non-blocking operations in the connection. these reads and writes are immediately returned, and return values that tell the caller how much work they could do. if this happens, it does not work at all, the program just needs to try again as soon as it knows that the connection is ready for more work. I changed the name to nb_frobnicate to indicate that this function does not require locking. - Most platforms provide a function that can abstract the loop through the connections, and checks to see if each one has data. This will be called either
select() or poll() (both may be available). However, I decided to show it this way, because you can work not only with a network connection. As far as I know, there can’t wait for any possible lock operation in the general case, if you also need to wait on the IO disk (only windows), timers, hardware interrupts (for example, screen updates), you need to manually interrogate different sources of events, sort of. - at the far end of the loop there is a
yield() function that sleeps until an IO interrupt happens. This is necessary if the IO is not ready for processing. If we didn’t do this, the program will wait, check and re-check each port and find out that there is no data ready for failures, unnecessary CPU losses. If the program only needs to wait for the network IO (or also the drive on posix systems), then select/poll could be called into lock mode, and it will sleep until the network connection is ready. If you have to wait for more events, you will have to use the non-blocking version of these functions, poll other sources of events according to their idiom, and then let it block once. - There is no equivalent to combining the flows from the previous example. Instead, it is similar to non-alignment. you can probably remove the listener from the poll collection, and then only the call to exit collection is empty.
- It is important that the entire program consists of simple loops and function calls. there it is not a question that carries the cost of a context switch. The downside of this is that the program is strictly sequential and cannot use multiple cores if they are present. In practice, this rarely occurs because most services will be IO limited by the speed at which connections become ready, rather than the cost of fictional
frobnicate() .
As I already mentioned, they can be used in combination, which allows you to use many advantages of several planning objects, such as more efficient use of CPUs on multi-core systems, as well as the benefits of non-blocking, i.e. lower load due to fewer context switches.
an example of this:
function worker(queue) connections = Collection<Connection>() while(running) while(queue not empty) connections.add(queue.pop) foreach connection in select(connections) if connection is open: nb_frobnicate(connection) else: connections.remove(connection) function main pool = new Collection<Thread, Queue<Connection> >() for index in num_cpus: pool[index] = new Thread(worker), new Queue<Connection> listener = new Listener(port) while(running) connection = listener.accept() selected_thread = 0 for index in pool.length: if pool[index].load() < pool[selected_thread].load() selected_thread = index pool[selected_thread].push(connection) pool[selected_thread].wake()
notes about this program:
- This creates a bunch of instances of the single-threaded version, but since they are multithreaded, they need to communicate with each other, somehow. this is done using
Queue<Connection> for each thread. - Also note that it uses the
nb_frobnicate handler for the same reason as a single-threaded program, and for the same reason, because each thread handles multiple connections. - The workflow pool is limited according to the number of processors that we have.
since there will be little benefit from the fact that there will be more expectations of flows than it is possible to run immediately. In practice, the optimal number of threads used can be applied to the application, but the number of processors is often reasonably good. - As before, the main thread accepts connections and passes them to worker threads. If a thread has already been executed with work on other threads, waiting for their readiness, then it will just sit there, even if a new connection is already ready. To facilitate this, the main thread then notifies the worker thread, waking it up, causing a lock to return. If this were a normal read, it would probably return a code error, but since we use
select() , it just returns an empty list of ready connections, and the thread can free its queue again.
Edit: Added code samples: