I am trying to deliver a large HTTP response using the Haskell Snap framework, but memory usage is increasing in proportion to the size of the response. Here are some cut-out test cases that use the big lazy ByteString:
import Snap.Core (Snap, writeLBS, readRequestBody) import Snap.Http.Server (quickHttpServe) import Control.Monad.IO.Class (MonadIO(liftIO)) import qualified Data.ByteString.Lazy.Char8 as LBS (ByteString, length, replicate) main :: IO () main = quickHttpServe $ site test1 where test1, test2 :: LBS.ByteString -> Snap () -- Send ss to client test1 = writeLBS -- Print ss to stdout upon receiving request test2 = liftIO . print site write = do body <- readRequestBody 1000 -- Making ss dependant on the request stops GHC from keeping a -- reference to ss as pointed out by Reid Barton. let bodyLength = fromIntegral $ LBS.length body write $ ss bodyLength ss c = LBS.replicate (1000000000000 * (c + 1)) 'S'
- test1 delivers ss to the client. Memory usage is growing in proportion to the size of the ByteString.
- test2 pushes ss on stdout to the Snap Monad stack after receiving the request. This runs in a small, constant amount of memory.
Responses are delivered using encoded encoding, so I thought Snap could also deliver the response in read-only memory. Is there any way to achieve this? It is also worth noting that the response begins to be delivered immediately. I tried using transformRequestBody but ran into the same problem.
Memory was measured using "top" on linux and was observed to constantly increase to 15 GB, at which point it was just starting to change. The request was completed, so the memory usage is a couple of orders of magnitude smaller than the size of the ByteString. As soon as the request was completed, he sat with a 15Gb resident. When I made another request, it still remained stable at 15 GB and completed the request as before. The virtual size remained within 5% of the resident size.
Dismissing 2 simultaneous requests on it first led to a drop in virtual and resident memory to about 5 GB, followed by an increase to about 17 GB, after which the machine became unusable, so I killed the process.
GHC Version 7.8.3
Snap version 0.14.0.5
source share