[FIXED] How to write responses of multiple asynchronous get requests to a single file in asynchronous httpclient java 11?

Issue

I can download a single media file using httpclient in java 11 like this

public class Httptest {
    
    private static HttpClient client = HttpClient.newBuilder().build();
            
    public static void main(String[] args) throws Exception {
        File fts = new File("P:/sample.ts");  //Destination of downloaded file
        fts.createNewFile();
        URI url = new URI("File url here"); //File Url
        
        HttpRequest request = HttpRequest.newBuilder()   //Creating HttpRequest using Builder class
                .GET()
                .uri(url)
                .build();
        Path file = Path.of("P:/samp.ts");
        //BodyHandlers class has methods to handle the response body
        // In this case, save it as a file (BodyHandlers.ofFile())
        HttpResponse<Path> response = client.send(request,BodyHandlers.ofFile(file)); 
    }
}

The above code snippet downloads the .ts file from the url. And it is downloaded properly.

Now, I have list of urls, List<URI> urls. I made asynchronous call to the list of urls, and also ensured concurrent calls by adding an Executor service. The place where I'm stuck is, how to write the list of responses to a single file.

The code I wrote so far:

public class httptest{
    
   // Concurrent requests are made in 4 threads
   private static ExecutorService executorService = Executors.newFixedThreadPool(4); 

   //HttpClient built along with executorservice
   private static HttpClient client = HttpClient.newBuilder() 
            .executor(executorService)
            .build();
    
   public static void main(String[] args) throws Exception{
        File fts = new File("P:/Spyder_directory/sample.ts");
        fts.createNewFile();
        List<URI> urls = Arrays.asList(
                         new URI("Url of file 1"),
                         new URI("Url of file 2"),
                         new URI("Url of file 3"),
                         new URI("Url of file 4"),
                         new URI("Url of file 5"));
        
        
        List<HttpRequest> requests = urls.stream()
                .map(HttpRequest::newBuilder)
                .map(requestBuilder -> requestBuilder.build())
                .collect(toList());
        Path file = Path.of("P:/Spyder_directory/sample.ts");
        List<CompletableFuture<HttpResponse<Path>>> results = requests.stream()
                .map(individual_req -> client.sendAsync(individual_req,BodyHandlers.ofFile(file)))
                .collect(Collectors.toList());
   }
}

The file sample.ts created at the end of execution does not have the response of the requests made. If you get the gist of my problem, can anyone suggest alternate solutions for this problem.

Solution

One possibility would be to use HttpResponse.BodyHandlers.ofByteArrayConsumer with a Consumer<Optional<byte[]>> that writes the bytes to a file. This would let you control how the file is opened, allowing you to append to an existing file rather than creating a new file each time.

Note that if you do that you should not use sendAsync because the requests will be sent concurrently, and the response will therefore be received concurrently too. If you still want to send the requests concurrently you will need to buffer the responses and impose some synchronization when writing them down to the file.

Answered By - daniel
Answer Checked By - Dawn Plyler (JavaFixing Volunteer)