[FIXED] Is this possible to apply bucketisation in Spring Boot (TOMCAT)

Issue

I exposed 2 api's

/endpoint/A and /endpoint/B .

@GetMapping("/endpoint/A")
    public ResponseEntity<ResponseA> controllerA() throws InterruptedException {

        ResponseA responseA = serviceA.responseClient();
        return ResponseEntity.ok().body(responseA);
    }



@GetMapping("/endpoint/B")
    public ResponseEntity<ResponseA> controllerB() throws InterruptedException {

        ResponseA responseB = serviceB.responseClient();
        return ResponseEntity.ok().body(responseB);
    }

Services implemented regarding endpoint A internally call /endpoint/C and endpoint B internally call /endpoint/D.

As external service /endpoint/D taking more time i.e getting response from /endpoint/A takes more time hence whole threads are stucked that is affecting /endpoint/B.
I tried to solve this using executor service having following implementation

@Bean(name = "serviceAExecutor")
    public ThreadPoolTaskExecutor serviceAExecutor(){

        ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
        taskExecutor.setCorePoolSize(100);
        taskExecutor.setMaxPoolSize(120);
        taskExecutor.setQueueCapacity(50);
        taskExecutor.setKeepAliveSeconds(120);
        taskExecutor.setThreadNamePrefix("serviceAExecutor");
        return taskExecutor;
    }

Even after implementing this if I received more than 200 request on /endpoint/A simultaneously (greater than default max number of threads in Tomcat server) then I am not getting responses from /endpoint/B as all threads are busy for getting response from endpoint A or in queue.

Can someone plz suggest is there any way to apply bucketization on each exposed endpoint level and allow only limited request to process at a time & put remaining into bucket/queue so that request on other endpoints can work properly ?

Edit:- following is solution approach

@GetMapping("/endpoint/A")
        public CompletableFuture<ResponseEntity<ResponseA>> controllerA() throws InterruptedException {
    
            return CompletableFuture.supplyAsync(()->controllerHelperA());
        }



    @GetMapping("/endpoint/B")
        public CompletableFuture<ResponseEntity<ResponseB>> controllerB() throws InterruptedException {
    
            return CompletableFuture.supplyAsync(()->controllerHelperB());
        }

private ResponseEntity<ResponseA> controllerHelperA(){

        ResponseA responseA = serviceA.responseClient();
        return ResponseEntity.ok().body(responseA);
    }

private ResponseEntity<ResponseB> controllerHelperB(){

        ResponseB responseB = serviceB.responseClient();
        return ResponseEntity.ok().body(responseB);
    }

Solution

Spring MVC supports the async servlet API introduced in Servlet API 3.0. To make it easier when your controller returns a Callable, CompletableFuture or DeferredResult it will run in a background thread and free the request handling thread for further processing.

@GetMapping("/endpoint/A")
public CompletableFuture<ResponseEntity<ResponseA>> controllerA() throws InterruptedException {
  return () {
    return controllerHelperA();
  }
}

private ResponseEntity<ResponseA> controllerHelperA(){

  ResponseA responseA = serviceA.responseClient();
  return ResponseEntity.ok().body(responseA);
}

Now this will be executed in a background thread. Depending on your version of Spring Boot and if you have configured your own TaskExecutor it will either

1. use the SimpleAsycnTaskExecutor (which will issue a warning in your logs),
2. the default provided ThreadPoolTaskExecutor which is configurable through the spring.task.execution namespace
3. Use your own TaskExecutor but requires additional configuration.

If you don't have a custom TaskExecutor defined and are on a relatively recent version of Spring Boot 2.1 or up (IIRC) you can use the following properties to configure the TaskExecutor.

spring.task.execution.pool.core-size=20
spring.task.execution.pool.max-size=120
spring.task.execution.pool.queue-capacity=50
spring.task.execution.pool.keep-alive=120s
spring.task.execution.thread-name-prefix=async-web-thread

Generally this will be used to execute Spring MVC tasks in the background as well as regular @Async tasks.

If you want to explicitly configure which TaskExecutor to use for your web processing you can create a WebMvcConfigurer and implement the configureAsyncSupport method.

@Configuration
public class AsyncWebConfigurer implements WebMvcConfigurer {

  private final AsyncTaskExecutor taskExecutor;

  public AsyncWebConfigurer(AsyncTaskExecutor taskExecutor) {
    this.taskExecutor=taskExecutor;
  }

  public void configureAsyncSupport(AsyncSupportConfigurer configurer) {
    configurer.setTaskExecutor(taskExecutor);
  }
}

You could use an @Qualifier on the constructor argument to specify which TaskExecutor you want to use.

Answered By - M. Deinum
Answer Checked By - Marilyn (JavaFixing Volunteer)