Base64 encoded string gets truncated via fgets call on IMAP parsing

Question

I parse emails using Zend_Mail, and, oddly enough, some content is truncated for no apparent reason and generates email details.

for example

Content-Disposition: attachment; filename="file.sdv" DQogICAgICBTT05FO0xBTkRJTkdTREE7U0FMR1NEQVRPIDtOQVNKIDtSRURTS0FQICAgICAgICAg ICAgIDsgRklTS0VTTEFHO1BSRVNFUlYgICA7ICBUSUxTVEFORDsgU1TYUlJFTFNFOyAgS1ZBTElU RVQ7T01TVFlQRSAgO01JTlNURVBSSVM7ICAgICBWRVJESTsgICBLVkFOVFVNOyAgUlVORFZFS1Qg IA0KLS0tLS0tLS0tLTstLS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS07LS0tLS0tLS0tLS0tLS0t LS0tLS07LS0tLS0tLS0tLTstLS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS0tLS0tLTstLS0tLS0t LS0tOy0tLS0tLS0tLTstLS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS0tLS0tLTstLS0tLS0tLS0t ICANCiAgICAgICAgIDA7MjAxMC4wOS4wODsyMDEwLjA5LjA4O05vcnNrO0dhcm4gICAgICAgICAg ICAgICAgOyAgICAgIDEwMjE7RkVSU0sgICAgIDsgICAgICAgMjEwOyAgIDQwMjA5OTk7ICAgICAg ICAyMDtFZ2Vub3ZlcnQ7ICAgICAgICAgIDsgICAzMDcyLDE2OyAgICAgICAyMTE7ICAgICAyNTMs MiAgDQogICAgICAgICAwOzIwMTAuMDkuMDg7MjAxMC4wOS4wODtOb3JzaztHYXJuICAgICAgICAg 

Truncates to

 Content-Disposition: attachment; filename="file.sdv" DQogICAgICBTT05FO0xBTkRJTkdTREE7U0FMR1NEQVRPIDtOQVNKIDtSRURTS0FQICAgICAgICAg ICAgIDsgRklTS0VTTEFHO1BSRVNFUlYgICA7ICBUSUxTVEFORDsgU1TYUlJFTFNFOyAgS1ZBTElU RVQ7T01TVFlQRSAgO01JTlNURVBSSVM7ICAgICBWRVJESTsgICBLVkFOVFVNOyAgUlVORFZFS1Qg IA0KLS0tLS0tLS0tLTstLS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS07LS0tLS0tLS0tLS0tLS0t LS0tLS07LS0tLS0tLS0tLTstLS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS0tLS0tLTstLS0tLS0t LS 

var_dump on each line shows this.

 string(78) "DQogICAgICBTT05FO0xBTkRJTkdTREE7U0FMR1NEQVRPIDtOQVNKIDtSRURTS0FQICAgICAgICAg " string(78) "ICAgIDsgRklTS0VTTEFHO1BSRVNFUlYgICA7ICBUSUxTVEFORDsgU1TYUlJFTFNFOyAgS1ZBTElU " string(78) "RVQ7T01TVFlQRSAgO01JTlNURVBSSVM7ICAgICBWRVJESTsgICBLVkFOVFVNOyAgUlVORFZFS1Qg " string(78) "IA0KLS0tLS0tLS0tLTstLS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS07LS0tLS0tLS0tLS0tLS0t " string(78) "LS0tLS07LS0tLS0tLS0tLTstLS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS0tLS0tLTstLS0tLS0t " string(5) "LS) " string(17) "TAG5 OK Success " 

or in another email

 DQogICAgICBTT05FO0xBTkRJTkdTREE7U0FMR1NEQVRPIDtOQVNKIDtSRURTS0FQICAgICAgICAg ICAgIDsgRklTS0VTTEFHO1BSRVNFUlYgICA7ICBUSUxTVEFORDsgU1TYUlJFTFNFOyAgS1ZBTElU RVQ7T01TVFlQRSAgO01JTlNURVBSSVM7ICAgICBWRVJESTsgICBLVkFOVFVNOyAgUlVORFZFS1Qg IA0KLS0tLS0tLS0tLTstLS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS07LS0tLS0tLS0tLS0tLS0t LS0tLS07LS0tLS0tLS0tLTstLS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS0tLS0tLTstLS0tLS0t LS0tOy0tLS0tLS0tLTstLS0tLS0tLS0tO 

I can’t understand why it stops there. Transfers should only be stopped at the end of the line. This is the string that receives the string from the IMAP server.

 $line = @fgets($this->_socket); 

The encoded text contains a type string, but again it is truncated in different parts in different letters.

 ----------;----------;----------;-----;--------------------;----------;----------;-- 

I tried adding size to fgets () but no results. I also turned on / off the setting of "auto_detect_line_endings" php_ini, again with no result.

I also opened an error report with ZF, although the error does not seem to be in the library.

Do you see something strange in this encoded string?

UPDATE

A new study shows that emails are truncated after 584 characters. Still don't know why. Submitted a question and Google. See here .

Bad email headers:

 Delivered-To: [e-mail] Received: by 10.216.3.208 with SMTP id 58cs248812weh; Fri, 20 Nov 2009 05:14:14 -0800 (PST) Received: by 10.204.153.217 with SMTP id l25mr1285471bkw.108.1258722853863; Fri, 20 Nov 2009 05:14:13 -0800 (PST) Return-Path: <> Received: from MTX4.mbn1.net (mtx4.mbn1.net [213.188.129.252]) by mx.google.com with SMTP id 2si1800716bwz.60.2009.11.20.05.14.12; Fri, 20 Nov 2009 05:14:13 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of MTX4.mbn1.net designates 213.188.129.252 as permitted sender) client-ip=213.188.129.252; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of MTX4.mbn1.net designates 213.188.129.252 as permitted sender) smtp.mail= Resent-From: < [e-mail] > Content-Type: multipart/mixed; boundary="===============1703099044==" MIME-Version: 1.0 From: < [e-mail] > To: < [e-mail] > CC: Subject: some subject Message-ID: < [e-mail] > X-OriginalArrivalTime: 20 Nov 2009 13:14:08.0121 (UTC) FILETIME=[5792C690:01CA69E3] Date: Fri, 20 Nov 2009 14:14:08 +0100 X-STA-Metric: 0 (engine=030) X-STA-NotSpam: tlf: vedlagt skip:__ 40 fil cc:2**0 X-STA-Spam: header:MIME-Version: charset:us-ascii header:Subject:1 to:2**0 header:From:1 X-BTI-AntiSpam: score:0,sta:0/030,dnsbl:passed,sw:off,bsn:38/passed,spf:off,bsctr:passed/1,dk:off,pbmf:none,ipr:0/3,trusted:no,ts:no,bs:no,ubl:passed X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply Resent-Message-Id: < [e-mail] > Resent-Date: Fri, 20 Nov 2009 14:14:11 +0100 (CET) --===============1703099044== Content-Type: application/octet-stream MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="file.sdv" DQpHUlVQUEVOQVZOICAgICAgICAgIDtLSthQRTtQUk9EQU5MO1BBS0tFTlI7TU9UVEFLTkFWTiAg ICAgICAgICAgICAgICAgICAgO1NPTjtMQU5ESU5HU0RBO1NBTEdTREFUTyA7TkFTSiA7UkVEU0tB UCAgIDtGSVNLRVNMQUcgO1BSRVNFUlYgICA7VElMU1RBTkQ7U1TYUlJFTFM7S1ZBTElURVQ7TUlO U1RFUFJJUzsgICAgICAgIFZFUkRJOyAgICAgS1ZBTlRVTTsgICAgUlVORFZFS1QgICAgDQotLS0t LS0tLS0tLS0tLS0tLS0tLTstLS0tLTstLS0tLS0tOy0tLS0tLS07LS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tOy0tLTstLS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS07LS0tLS0tLS0tLTst LS0tLS0tLS0tOy0tLS0tLS0tLS07LS0tLS0tLS07LS0tLS0tLS07LS0tLS0tLS07LS0tLS0tLS0t LTstLS0tLS0tLS0tLS0tOy0tLS0tLS0tLS0tLTstLS0tLS0tLS0tLS0gICAgDQpMb3JlbnR6ZW4g .... 

For those interested in the answer, and not in the (ex) bounty, there are more tips.

Gmail returns a short value in response to RFC822.SIZE, which can lead to truncated messages. (They are disabled by one byte for each header line, apparently not counting two characters for CR / LF.)

Answer 1

I think you are looking for the wrong place.

The imap server gives you a truncated message, and then returns the TAG5 OK Success status TAG5 OK Success .

I don’t see how your processing (/ php) of the socket will lead to the disappearance of a stream of several kilobytes in order to magically correct the stream right before this status line.

Thus, either the message is truncated by itself (did you check the contents of the message in any other way?), Or the imap server is simply broken.

The first things I would do are:

• find a quiet enough environment to host your project, where you can strace -f -s 10240 -p <pid> apache process to check socket interaction (assuming that in linux / apache environment)
• and / or: use tcpdump , ethereal or equivalent to check what is on the line

I assume that you will see exactly the same truncated lines that go into the wire. This means that you can switch focus to the imap server.

Satisfying yourself with what you are looking for in the right place can save you a lot of time.

Answer 2

1: try removing @ for more details

2: try using http://www.php.net/manual/en/function.fread.php instead of fgets

This may have something to do with the IMAP server, because I see TAG5 OK Success as an answer, even if it should not be there.

Answer 3

Have you tried to release another fgets and see if you will receive the rest of the information? You may receive a multi-component email that will require multiple requests.

But regardless of whether you use functions designed to access files on the network. This usually works fine, but problems may occur depending on the network. For example, you can use file_get_contents to retrieve a web page. But if the problem issues a redirect, then it fails. But using curl will be much more successful.

If you really want to read the network socket, try socket_read. It is designed with the net in mind, like curl.

Answer 4

I do not know Zend and forgot about PHP, but played with MIME and HTTP before (C ++).

I suggest you start looking for a way to add a Content-Length header. It gives a message decoder / message loader hint to expect a specific size in the content (message payload). (Not sure if IMAP does this)

In the above code, I will try to convince fgets to read a certain amount of expected data from the network. Perhaps the data is buffered or not yet sent over the network (asynchronous communication), and fgets only reads the internal buffer, thus stopping before the entire message has been read.

• To find out if this is true, send a short message that falls under “584 breakpoint”.
• Trace the network to see if all data flows. (You probably need to do local configuration)

The code you are linking to is here ?

Answer 5

Most likely, one of your server hardware is compromised and therefore you want to completely change it or just change the RAM or Disk-Drives modules. I have experience with encoding based on Web and Mail, and I can confirm that the base64 encoded string is very safe. At least it uses a texture mapping algorithm.