How to convert (StorableArray (Int, Int) Word8) to lazy ByteString?

Question

I am trying to download a PNG file, get uncompressed RGBA bytes, and then send them to gzip or zlib packages.

The pngload package returns image data as (StorableArray (Int, Int) Word8), and compression packets take lazy bytes. So I'm trying to create a function (StorableArray (Int, Int) Word8 -> ByteString).

So far I have tried the following:

import qualified Codec.Image.PNG as PNG import Control.Monad (mapM) import Data.Array.Storable (withStorableArray) import qualified Data.ByteString.Lazy as LB (ByteString, pack, take) import Data.Word (Word8) import Foreign (Ptr, peekByteOff) main = do -- Load PNG into "image"... bytes <- withStorableArray (PNG.imageData image) (bytesFromPointer lengthOfImageData) bytesFromPointer :: Int -> Ptr Word8 -> IO LB.ByteString bytesFromPointer count pointer = LB.pack $ mapM (peekByteOff pointer) [0..(count-1)] 

This causes the stack to run out of memory, so I'm doing something very wrong. There are more things that I could try with Ptr and ForeignPtr, but there are many "unsafe" functions there.

Any help here would be greatly appreciated; I'm pretty puzzled.

Answer 1

Packing and unpacking is generally a bad idea for performance. If you have Ptr and the length in bytes, you can create a strict byte setting in two ways:

• packCStringLen
• unsafePackCStringLen

Like this:

 import qualified Codec.Image.PNG as PNG import Control.Monad import Data.Array.Storable (withStorableArray) import Codec.Compression.GZip import qualified Data.ByteString.Lazy as L import qualified Data.ByteString.Unsafe as S import Data.Word import Foreign -- Pack a Ptr Word8 as a strict bytestring, then box it to a lazy one, very -- efficiently bytesFromPointer :: Int -> Ptr Word8 -> IO L.ByteString bytesFromPointer n ptr = do s <- S.unsafePackCStringLen (castPtr ptr, n) return $! L.fromChunks [s] -- Dummies, since they were not provided image = undefined lengthOfImageData = 10^3 -- Load a PNG, and compress it, writing it back to disk main = do bytes <- withStorableArray (PNG.imageData image) (bytesFromPointer lengthOfImageData) L.writeFile "foo" . compress $ bytes 

I am using the O (1) version, which simply repackages Ptr from StorableArray. You might want to copy it first through "packCStringLen".

Answer 2

The problem with your "bytesFromPointer" is that you take the packaged view, StorableArray from pngload, and want to convert it to another packed view, ByteString, passing through an intermediate list. Sometimes laziness means that an intermediate list will not be built in memory, but this is not so.

The mapM function is the first intruder. If you expand mapM (peekByteOff pointer) [0..(count-1)] , you will get

 el0 <- peekByteOff pointer 0 el1 <- peekByteOff pointer 1 el2 <- peekByteOff pointer 2 ... eln <- peekByteOff pointer (count-1) return [el0,el1,el2,...eln] 

since all these actions take place in the IO monad, they are performed in order. This means that each element of the output list must be constructed before the list is created, and laziness can never help you.

Even if the list was created lazily, as Don Stewart notes, the "pack" function will still ruin your work. The problem with the “package” is that it needs to know how many items in the list the correct amount of memory allocates. To find the length of the list, the program must go through it to the end. Due to the need to calculate the length, the list must be fully loaded before it can be packed into a byte string.

I consider "mapM" as well as "pack" as the smell of code. Sometimes you can replace mapM with "mapM_", but in this case it is better to use byte creation functions, for example. "PackCStringLen".