Would a twisted good choice for creating a multi-threaded server?

Question

I need to collect hundreds of pop3 email accounts and I want to create a reliable server for this.

Would twisting be a good choice for this type of project?

Right now, the simplest prototype is to pull pop3 out of one account, and then pull out of many, but this will be a serialized process.

I want to create a server with multiple threads so that it can do something at the same time.

Answer 1

Twisted is an event-driven networking infrastructure written in Python. It is heavily dependent on asynchronous and non-blocking functions and is best suited for developing network applications that use them. It supports streaming support for use when you cannot provide asynchronous non-blocking I / O. This is based on the fact that most of the time is spent in network I / O operations.

Two models that use this are a stream model in which you create multiple threads, each of which performs a single task or a single process that uses non-blocking I / O to perform multiple tasks in a single process by interleaving several tasks. Twisted is very suitable for the second model.

Non-blocking model

+--------------------------+ |task1 | wait period | comp| +--------------------------+ +--------------------------+ |task2 | wait period | comp| +--------------------------+ 

You can create a very reliable server with Twisted and have POP3 / IMAP support.

There is an example of creating a pop3 client with twisted .

Answer 2

Given that most of your POP3 activity will be network I / O, Twisted will straighten out here. You actually do not have as many threads as performing event-based asynchronous socket operations, which is Twisted's crown of fame.

So yes, Twisted would be a good choice for this type of project. It can work with clients and server equally well, and it is almost trivial to deploy a new asynchronous TCP client, and it already has the POP3 TCP Client available by default.

Answer 3

This is a good choice for a server, but from your description you are actively looking for a multi-threaded POP client.

Twisted in order to respond to events such as incoming requests, you need to send requests, so in this case I am afraid that the twisted has limited value.

Answer 4

A word of caution with twisted, while twisted very durable, I found that deploying one hundred threads using the code examples available in the documentation is a recipe for race conditions and dead ends. My suggestion is to try twisting, but in case the twisted becomes unmanageable, there is a multi-threaded stdlib module waiting in the wings. I had good success with the consumer-producer model using the above library.