[FIXED] Spring boot Java heap space for downloading large files

Issue

I am trying to write a REST api to allow users to download large files (ie > 2GB) on Spring boot. I am hitting with "Java Heap outOfMemoryException". I tried to triage the issue, i see that HttpServetResponse object is of type : ContentCachingResponseWrapper. This class caches all content written to the output stream and when cached data size becomes around 258MB, i get OutOfMemoryException. Why at 248 MB, because JVM has 256 MB of heap memory.

The default flushBuffer() method in ContentCachingResponseWrapper is empty. If i try to call copyBodyToResponse(), which is used to copy data from cache to stream, it works fine, but it closes the stream as well. This leads to only sending first chunk of data to client.

any suggestions ?

public void myDownloader(HttpServletRequest request, HttpServletResponse response) {
             //response.getClass() is: ContentCachingResponseWrapper  
             byte[] buffer = new byte[1048576];  // 1 MB Chunks
             FileInputStream inputStream = new FileInputStream(PATH_TO_SOME_VALID_FILE);
             int bytesRead= 0;
             ServletOutputStream outputStream = response.getOutputStream();

             while ((bytesRead = inputStream.read(buffer)) != -1) {              
                 outputStream.write(buffer, 0, bytesRead);
                 outputStream.flush();
                 response.flushBuffer();
               } 
}

I get following error:

Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.springframework.util.FastByteArrayOutputStream.addBuffer(FastByteArrayOutputStream.java:303) ~[spring-core-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
    at org.springframework.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:118) ~[spring-core-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
    at org.springframework.web.util.ContentCachingResponseWrapper$ResponseServletOutputStream.write(ContentCachingResponseWrapper.java:239) ~[spring-web-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]

Solution

This is a very basic way to download the file, but if you call it from a browser, the browser will display it on screen, and may spin forever (browser problem if you ask me):

@RequestMapping(path = "/downloadLargeFile", method = RequestMethod.GET)
public ResponseEntity<Resource> downloadLargeFile() {
    final File file = new File("c:/large.bin");
    final FileSystemResource resource = new FileSystemResource(file);
    return ResponseEntity.ok().body(resource);
}

So you can include some headers with info about the file, and the browser downloads it to a file in your Downloads directory, and does not spin:

@RequestMapping(path = "/downloadLargeFile2", method = RequestMethod.GET)
public ResponseEntity<Resource> downloadLargeFile2() {
    final HttpHeaders httpHeaders = new HttpHeaders();
    final File file = new File("c:/large.bin");
    final FileSystemResource resource = new FileSystemResource(file);
    httpHeaders.set(HttpHeaders.LAST_MODIFIED, String.valueOf(file.lastModified()));
    httpHeaders.set(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + file.getName() + "\"");
    httpHeaders.set(HttpHeaders.CONTENT_LENGTH, String.valueOf(file.length()));
    return ResponseEntity.ok()
        .headers(httpHeaders)
        .contentLength(file.length())
        .contentType(MediaType.APPLICATION_OCTET_STREAM)
        .body(resource);
}

To chunk the response, use InputStreamResource, and apparently Spring closes the InputStream:

@RequestMapping(path = "/pub/storage/downloadLargeFile4", method = RequestMethod.GET)
public ResponseEntity<InputStreamResource> downloadLargeFile4()
    throws Exception {
    final HttpHeaders httpHeaders = new HttpHeaders();
    final File file = new File("c:/large.bin");
    final InputStream inputStream = new FileInputStream(file);
    final InputStreamResource resource = new InputStreamResource(inputStream);
    httpHeaders.set(HttpHeaders.LAST_MODIFIED, String.valueOf(file.lastModified()));
    httpHeaders.set(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + file.getName() + "\"");
    httpHeaders.set(HttpHeaders.CONTENT_LENGTH, String.valueOf(file.length()));
    return ResponseEntity.ok()
        .headers(httpHeaders)
        .contentLength(file.length())
        .contentType(MediaType.APPLICATION_OCTET_STREAM)
        .body(resource);
}

Imports:

import org.springframework.core.io.FileSystemResource;
import org.springframework.core.io.Resource;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

I also find this thread to be useful: download a file from Spring boot rest service

Answered By - ShortPasta
Answer Checked By - Candace Johnson (JavaFixing Volunteer)