PHP receives over 20,000 IMAP emails

Question

I am trying to export multiple mailboxes to a database. My current script will attach IMAP and just connect all messages. Although with large mailboxes this will not work, and it will slow down or even stop.

The idea is to run a script daily to “copy” all messages that are not already in the database in the database. What is the best way to get a large number of emails (20 thousand. Letters extends to approximately 40-50 folders).

In the end, it will need to work from one server to scan hundreds or even thousands of accounts daily (so imagine the amount of data). It will store mail (uid and subject) in the database and create a package that will be stored on dataserver (so it also needs to receive attachments).

Answer 1

So you want to back up your e-mail via IMAP. These are professional software tools that do this.

Let's start with something simple: download email for one specific user from your inbox. To do this, you must (a) log in with the user credentials, (b) select the INBOX folder and (c) download the message (suppose you already know its UID, which is 55). You do this in IMAP as follows (only requests: no answers shown):

01 LOGIN username password 02 SELECT INBOX 03 UID FETCH 55 BODY[] 

Each message in a specific folder is assigned a UID . This is a unique identifier for a message that never changes - it cannot be used by any other message in this folder. New messages must have a higher UID than previous ones. This makes it a useful tool for determining whether you have already downloaded a message earlier.

Next step: let's now look at loading all new messages into the INBOX folder. Suppose you download messages for the first time, and INBOX currently has messages with UIDs 54, 55, and 57. You can download these messages at the same time using a command, for example:

 03 UID FETCH 54,55,57 BODY[] 

(You might want to split it in batches (e.g. 30 at a time) if there is a lot to download.) After that, you save the highest UID that you have downloaded so far. Next time you can check the UID higher than this:

 04 UID FETCH 58:* UID 

This will receive the UID (only) for messages with UIDs with 58 or more. If you get the results, you download them and save the UID again. And so on.

There is one catch. Message UIDs are valid until the folder’s UIDVALIDITY attribute (included in response to the SELECT command) is changed. If this changes for any reason, the folder is invalid and you need to download all messages to this folder again and again.

Finally, you want to expand this to work for all folders for all users. To get all the folders for a specific user, you use the IMAP LIST command:

 05 LIST "" "*" 

You will need to know the credentials for users in advance and sort them out.

This is the IMAP theory that underlies what you need to do. Implementing this in PHP remains as an exercise.

Answer 2

Are you imap_ping ?

imap_ping () pings the stream to see if it is all active. This may open new mail; this is the preferred method for periodic "new mail check", as well as "keep alive" for servers that have an idle timeout.

Others to look at: imap_timeout imap_reopen

In fact, there is a method called reopen that suggests something is wrong :)

Another option that comes to mind if you simply cannot save the connection is to export the data to the mbox format and get it locally. It can be faster for a huge mailbox and can fix timeout / connection problems.