[FIXED] Spring WebClient corrupts binary data

Issue

I'm sending files containing binary data from service A to service B. When the number of files is relatively small (let's say 5) everything works well. However, when I try to send more files (let's say several hundred) it sometimes fails. I tried to check what is happening with this binary data, and it looks like WebClient corrupts it in some way (weird padding appears at the end).

I created a minimal reproducible example to illustrate this issue.

Endpoint in service B (consuming binary files):

@RestController
class FilesController {

    @PostMapping(value = "/files")
    Mono<List<String>> uploadFiles(@RequestBody Flux<Part> parts) {
        return parts
                .filter(FilePart.class::isInstance)
                .map(FilePart.class::cast)
                .flatMap(part -> DataBufferUtils.join(part.content())
                        .map(buffer -> {
                            byte[] data = new byte[buffer.readableByteCount()];
                            buffer.read(data);
                            DataBufferUtils.release(buffer);
                            return Base64.getEncoder().encodeToString(data);
                        })
                )
                .collectList();
    }
}

Tests illustrating how the service A sends data:

public class BinaryUploadTest {

    private final CopyOnWriteArrayList<String> sentBytes = new CopyOnWriteArrayList<>();

    @BeforeEach
    void before() {
        sentBytes.clear();
    }

    /**
     * this test passes all the time
     */
    @Test
    void shouldUpload5Files() {
        // given
        MultiValueMap<String, HttpEntity<?>> body = buildResources(5);

        // when
        List<String> receivedBytes = sendPostRequest(body);

        // then
        assertEquals(sentBytes, receivedBytes);
    }

    /**
     * this test fails most of the time
     */
    @Test
    void shouldUpload1000Files() {
        // given
        MultiValueMap<String, HttpEntity<?>> body = buildResources(1000);

        // when
        List<String> receivedBytes = sendPostRequest(body);

        // then
        assertEquals(sentBytes, receivedBytes);
    }

    private List<String> sendPostRequest(MultiValueMap<String, HttpEntity<?>> body) {
        return WebClient.builder().build().post()
                .uri("http://localhost:8080/files")
                .contentType(MediaType.MULTIPART_FORM_DATA)
                .body(BodyInserters.fromMultipartData(body))
                .retrieve()
                .bodyToMono(new ParameterizedTypeReference<List<String>>() {
                })
                .block();
    }

    private MultiValueMap<String, HttpEntity<?>> buildResources(int numberOfResources) {
        MultipartBodyBuilder builder = new MultipartBodyBuilder();
        for (int i = 0; i < numberOfResources; i++) {
            builder.part("item-" + i, buildResource(i));
        }
        return builder.build();
    }

    private ByteArrayResource buildResource(int index) {
        byte[] bytes = randomBytes();
        sentBytes.add(Base64.getEncoder().encodeToString(bytes)); // keeps track of what has been sent
        return new ByteArrayResource(bytes) {
            @Override
            public String getFilename() {
                return "filename-" + index;
            }
        };
    }

    private byte[] randomBytes() {
        byte[] bytes = new byte[ThreadLocalRandom.current().nextInt(16, 32)];
        ThreadLocalRandom.current().nextBytes(bytes);
        return bytes;
    }
}

What could be the reason for this data corruption?

Solution

It turned out to be a bug in the Spring Framework (in the MultipartParser class to be more precise). I have created a GitHub issue which will be fixed in the next version (5.3.16). The bug is fixed by this commit.

Answered By - k13i
Answer Checked By - Pedro (JavaFixing Volunteer)