Haskell: lazily readable binary with binary

Question

I am trying to read in a binary file and parse it lazily using a “binary” package. The package documentation gives an example of how to do this without forcing all the input for a script very similar to mine:

 example2 :: BL.ByteString -> [Trade]
 example2 input
  | BL.null input = []
  | otherwise =
    let (trade, rest, _) = runGetState getTrade input 0
    in trade : example2 rest

However, this uses an obsolete function runGetState, which itself points to a function runGetIncremental.

The problem is that the 'runGetIncremental' function apparently forces the remaining input to be a strict bytestring, causing it to load the entire file into memory. In fact, I see a memory usage of about 6 GB when I try to run this. Even the implementation runGetStatenow seems to be based on runGetIncremental, and then returns a lowercase bytestring back to lazy using chunk.

, , ? , ? , , .

Answer 1

, pipes-binary pipes-bytestring. :

import Control.Monad (void)
import Data.Binary
import Pipes
import Pipes.Binary (decodeMany)
import Pipes.ByteString (fromHandle)
import qualified Pipes.Prelude as P
import System.IO

decodeHandle :: (Binary a) => Handle -> Producer a IO ()
decodeHandle handle = void $ decodeMany (fromHandle handle) >-> P.map snd

void map snd , decodeMany (, ). , .

, decodeHandle, Trade, :

data Trade = Trade

instance Binary Trade where
    get   = return Trade
    put _ = return ()

instance Show Trade where show _ = "Trade"

main = withFile "inFile.txt" ReadMode $ \handle -> runEffect $
    for (decodeHandle handle) $ \trade -> do
        lift $ print (trade :: Trade)
        -- do more with the parsed trade

for , :

main = withFile "inFile.txt" ReadMode $ \handle -> runEffect $
    decodeHandle handle >-> P.print

, . , take , , :

main = withFile "inFile.txt" ReadMode $ \handle -> runEffect $
    for (decodeHandle handle >-> P.take 4) $ \trade -> do
        ... -- This will only process the first 4 trades

-- or using purely pipe composition:

main = withFile "inFile.txt" ReadMode $ \handle -> runEffect $
    decodeHandle handle >-> P.take 4 >-> P.print