I am developing a program that allows you to simulate networks on one machine. For this, I use Twisted for asynchronous I / O, as there may be a bit of thread for each “connection”. (I also implemented a similar program in Java using their NIO). However, as the scale of the emulated network increases, the bandwidth on Twisted decreases. Comparing this to a Java implementation, at the same network level, Java throughput continues to grow. (The growth rate is slowing, but it is still increasing). For instance. (Python 100 nodes = 58 MB Total throughput, 300 nodes = 45 MB, Java 100 nodes = 24 MB, 300 nodes = 56 MB).
I am wondering if anyone has any suggestions on why this might happen?
The only reason I can think of is because Java has every peer working in its own thread (which contains its own selector that controls peer-to-peer connections). In the python version, everything is registered in the reactor (and then in a single selector). Because python is scalable, a single selector cannot respond so fast. However, this is just an assumption that if someone has more specific information, he will be assigned.
EDIT: I did some testing, as suggested by Jean-Paul Calderon, the results are published in imgur . For those who might wonder that the following Avg bandwidth was indicated for tests. (Profiling was performed using cProfile, tests were completed within 60 seconds)
Epoll Reactor: 100 Peers: 20.34 MB, 200 Peers: 18.84 MB, 300 Peers: 17.4 MB
Select reactor: 100 Peers: 18.86 MB, 200 Peers: 19.08 MB, 300 Peers: 16.732 MB
A couple of things that seemed to rise and fall with the reported bandwidth were calls made on main.py:48(send), but this procedure is not unexpected, since data is sent here.
For both reactors, the time spent on the send function on the socket increased as the throughput decreased, and the number of calls to the send function decreased as the throughput decreased. (That is: more time was spent sending to the socket, with fewer calls to send over the socket.) For example. 2.5 seconds for epoll {method of "sending" _socket.socket objects} to 100 peers for 413600 calls, up to 5.5 seconds for epoll to 300 peers for 354300 calls.
To try to answer the original question, does this data indicate that the selector is a limiting factor? The time spent on the selector seems to decrease as the number of peers increases (if the selector slowed everything down, would you expect the time spent inside to go up?) Is there anything else that could slow down the amount of data sent ? (Sending data is just one function for each peer user that logs over and over in the .calllater reactor. This is main.py:49 (send))