[FIXED] Entire large json response is not returned

Issue

I am trying to get a large json response using a recursive REST call like below:

private List<MyPojo> recursiveRestCallMethod(String folderId) throws IOException {
    
    List<MyPojo> mypojoList = new ArrayList<>();
    
    String hugeJson = webClient.get()
            .uri("/my/rest/api/accepting/" + folderId
                    + "/and/producing/huge/jsonresponse/for/all/files/recursively")
            .retrieve().bodyToMono(String.class).block();
    
    byte[] bytes = hugeJson.getBytes("UTF-8");
    
    String json = new String(bytes, StandardCharsets.UTF_8);
    
    ObjectMapper objectMapper = new ObjectMapper();
    
    ObjectNode node = objectMapper.readValue(json, ObjectNode.class);
    
    objectMapper.configure(
            DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
    
    if (node.get("list").get("entries").isArray()) {
        for (JsonNode jsonNode : node.get("list").get("entries")) {
            
            MyPojo pojo = new MyPojo();
            
            JsonNode mainNode = jsonNode.get("entry");
            
            if (mainNode.get("isFile").asBoolean()) {
                JsonNode nameNode = mainNode.get("name");
                pojo.setNodename(nameNode.toString());
                // and 20 more fields
                mypojoList.add(pojo);
            }
            if (mainNode.get("isFolder").asBoolean()) {
                mypojoList.addAll(recursiveRestCallMethod(mainNode.get("id").toString().replaceAll("\"", "").trim()));
            }
            
        }
        return mypojoList;
    }
    return null;
}

Now everytime the json returned has 4193150 characters and it throws exception - Unexpected end-of-input: expected close marker for Object as reported here and some other SO threads (obviously, the json is not complete and valid).

The incomplete json I am getting looks something like:

{"list":{"pagination":{"count":6097,"hasMoreItems":false,"totalItems":6097,"skipCount":0,"maxItems":10000},"entries":[{"entry":{"....

From above, as you can see I should get 6097 objects, but I am getting only 2024 entry array items. And after that json ends abruptly. i.e. invalid json string.

However, for smaller response, where I have 20/30 entry array items, it works as expected.

Note: I am using Spring-Boot 2.4.5 and hence Jackson 2.12.4

Question: Even though I am using .block(), why the response stops at 4193150 characters? What I am doing wrong here?

Solution

Not sure what was wrong using String but when I switched to DataBuffer, it worked fine.

Here is the snippet for what I used:

    final Flux<DataBuffer> hugeJson = webClient.get()
            .uri("/my/rest/api/accepting/" + folderId
                    + "/and/producing/huge/jsonresponse/for/all/files/recursively")
            .accept(MediaType.ALL)
            .retrieve()
            .bodyToFlux(DataBuffer.class);

Answered By - Ajay Kumar
Answer Checked By - Candace Johnson (JavaFixing Volunteer)