Download large file from server using Java Spring MVC REST template

Question

I have a REST service that sends me a large ISO file, there are no problems with the REST service. Now I wrote a web application that calls the rest service to get the file, on the client side (web application) I get Out Of Memory Exception.Below is my code

HttpHeaders headers = new HttpHeaders();//1 Line headers.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM));//2 Line headers.set("Content-Type","application/json");//3 Line headers.set("Cookie", "session=abc");//4 Line HttpEntity statusEntity=new HttpEntity(headers);//5 Line String uri_status=new String("http://"+ip+":8080/pcap/file?fileName={name}");//6 Line ResponseEntity<byte[]>resp_status=rt.exchange(uri_status, HttpMethod.GET, statusEntity, byte[].class,"File5.iso");//7 Line 

I get a memory exception in line 7, I think I will have to buffer and get the parts, but I don’t know how I can get this file from the server, the file size is from 500 to 700 MB. Someone can help.

Exceptional Stack:

  org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778) javax.servlet.http.HttpServlet.service(HttpServlet.java:622) javax.servlet.http.HttpServlet.service(HttpServlet.java:729) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) root cause java.lang.OutOfMemoryError: Java heap space java.util.Arrays.copyOf(Arrays.java:3236) java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113) org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164) org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:58) org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:1) org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:153) org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:81) org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:627) org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:1) org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:454) org.springframework.web.client.RestTemplate.execute(RestTemplate.java:409) org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:385) com.pcap.webapp.HomeController.getPcapFile(HomeController.java:186) 

The server side REST service code, which works fine,

 @RequestMapping(value = URIConstansts.GET_FILE, produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE}, method = RequestMethod.GET) public void getFile(@RequestParam(value="fileName", required=false) String fileName,HttpServletRequest request,HttpServletResponse response) throws IOException{ byte[] reportBytes = null; File result=new File("/home/arpit/Documents/PCAP/dummyPath/"+fileName); if(result.exists()){ InputStream inputStream = new FileInputStream("/home/arpit/Documents/PCAP/dummyPath/"+fileName); String type=result.toURL().openConnection().guessContentTypeFromName(fileName); response.setHeader("Content-Disposition", "attachment; filename=" + fileName); response.setHeader("Content-Type",type); reportBytes=new byte[100];//New change OutputStream os=response.getOutputStream();//New change int read=0; while((read=inputStream.read(reportBytes))!=-1){ os.write(reportBytes,0,read); } os.flush(); os.close(); } 

Answer 1

This is how I do it. Based on the tips from this Spring Jira issue.

 RestTemplate restTemplate // = ...; // Optional Accept header RequestCallback requestCallback = request -> request.getHeaders() .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL)); // Streams the response instead of loading it all in memory ResponseExtractor<Void> responseExtractor = response -> { // Here I write the response to a file but do what you like Path path = Paths.get("some/path"); Files.copy(response.getBody(), path); return null; }; restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor); 

From Jira's above question:

Please note that you cannot simply return an InputStream from the extractor, because by the time the execute method returns, the base connection and stream are already closed.

Update for Spring 5

Spring 5 introduced the WebClient class that allows asynchronous (e.g. non-blocking) http requests. From the doc:

Compared to RestTemplate, WebClient is:

• Does not block, reacts and supports higher parallelism with less hardware resources.
• provides a functional API that takes advantage of the Java 8 lambda code.
• supports both synchronous and asynchronous scripts.
• supports streaming up or down from the server.

Answer 2

This prevents the entire request from loading into memory.

 SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory(); requestFactory.setBufferRequestBody(false); RestTemplate rest = new RestTemplate(requestFactory); 

For java.lang.OutOfMemoryError: Java heap of space can be solved by adding more memory to the JVM:

-Xmxn Specifies the maximum size of the memory allocation pool in bytes. This value must be a multiple of 1024 greater than 2 MB. Add the letters k or K to indicate kilobytes, or m or M to indicate megabytes. The default value is selected at run time based on the system configuration.

For server deployment, -Xms and -Xmx are often set to the same value. See Garbage Collector Ergonomics at http://docs.oracle.com/javase/7/docs/technotes/guides/vm/gc-ergonomics.html

Examples:

-Xmx83886080
-Xmx81920k
-Xmx80m

Your problem is probably not related to the request that you are trying to execute (upload a large file), but there is not enough memory for this process.

Answer 3

The best version of the above correct answer might be the following code. This method sends a download request to another application or service, acting as the actual source of truth for the downloaded information.

 public void download(HttpServletRequest req, HttpServletResponse res, String url) throws ResourceAccessException, GenericException { try { logger.info("url::" + url); if (restTemplate == null) logger.info("******* rest template is null***********************"); RequestCallback requestCallback = request -> request.getHeaders() .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL)); // Streams the response instead of loading it all in memory ResponseExtractor<ResponseEntity<InputStream>> responseExtractor = response -> { String contentDisposition = response.getHeaders().getFirst("Content-Disposition"); if (contentDisposition != null) { // Temporary location for files that will be downloaded from micro service and // act as final source of download to user String filePath = "/home/devuser/fileupload/download_temp/" + contentDisposition.split("=")[1]; Path path = Paths.get(filePath); Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING); // Create a new input stream from temporary location and use it for downloading InputStream inputStream = new FileInputStream(filePath); String type = req.getServletContext().getMimeType(filePath); res.setHeader("Content-Disposition", "attachment; filename=" + contentDisposition.split("=")[1]); res.setHeader("Content-Type", type); byte[] outputBytes = new byte[100]; OutputStream os = res.getOutputStream(); int read = 0; while ((read = inputStream.read(outputBytes)) != -1) { os.write(outputBytes, 0, read); } os.flush(); os.close(); inputStream.close(); } return null; }; restTemplate.execute(url, HttpMethod.GET, requestCallback, responseExtractor); } catch (Exception e) { logger.info(e.toString()); throw e; } } 

Answer 4

You must use a multi-page file attachment, so the file stream does not load into memory. In this example, I am using the leisure service implemented with Apache CXF.

 ... import org.apache.cxf.jaxrs.ext.multipart.Attachment; ... @Override @Path("/put") @Consumes("multipart/form-data") @Produces({ "application/json" }) @POST public SyncResponseDTO put( List<Attachment> attachments) { SyncResponseDTO response = new SyncResponseDTO(); try { for (Attachment attr : attachments) { log.debug("get input filestream: " + new Date()); InputStream is = attr.getDataHandler().getInputStream();