Issue
We have a bunch of microservices based on Spring Boot 2.5.4 also including spring-kafka:2.7.6 and spring-boot-actuator:2.5.4. All the services use Tomcat as servlet container and graceful shutdown enabled. These microservices are containerized using docker.
Due to a misconfiguration, yesterday we faced a problem on one of these containers because it took a port already bound from another one.
Log states:
Stopping service [Tomcat]
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
***************************
APPLICATION FAILED TO START
***************************
Description:
Web server failed to start. Port 8080 was already in use.
However, the JVM is still running, because of the kafka consumers/streams.
I need to destroy everything or at least do a System.exit(error-code) to trigger the docker restart policy. How I could achieve this? If possible, a solution using configuration is better than a solution requiring development.
I developed a minimal test application made of the SpringBootApplicationand a KafkaConsumer class to ensure the problem isn't related to our microservices. Same result.
POM file
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.5.4</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
...
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
Kafka listener
@Component
public class KafkaConsumer {
@KafkaListener(topics = "test", groupId = "test")
public void process(String message) {
}
}
application.yml
spring:
kafka:
bootstrap-servers: kafka:9092
Log file
2021-12-17 11:12:24.955 WARN 29067 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.PortInUseException: Port 8080 is already in use
2021-12-17 11:12:24.959 INFO 29067 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2021-12-17 11:12:24.969 INFO 29067 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-12-17 11:12:24.978 ERROR 29067 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
Web server failed to start. Port 8080 was already in use.
Action:
Identify and stop the process that's listening on port 8080 or configure this application to listen on another port.
2021-12-17 11:12:25.151 WARN 29067 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-test-1, groupId=test] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE}
2021-12-17 11:12:25.154 INFO 29067 --- [ntainer#0-0-C-1] org.apache.kafka.clients.Metadata : [Consumer clientId=consumer-test-1, groupId=test] Cluster ID: NwbnlV2vSdiYtDzgZ81TDQ
2021-12-17 11:12:25.156 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-test-1, groupId=test] Discovered group coordinator kafka:9092 (id: 2147483636 rack: null)
2021-12-17 11:12:25.159 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-test-1, groupId=test] (Re-)joining group
2021-12-17 11:12:25.179 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-test-1, groupId=test] (Re-)joining group
2021-12-17 11:12:27.004 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-test-1, groupId=test] Successfully joined group with generation Generation{generationId=2, memberId='consumer-test-1-c5924ab5-afc8-4720-a5d7-f8107ace3aad', protocol='range'}
2021-12-17 11:12:27.009 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-test-1, groupId=test] Finished assignment for group at generation 2: {consumer-test-1-c5924ab5-afc8-4720-a5d7-f8107ace3aad=Assignment(partitions=[test-0])}
2021-12-17 11:12:27.021 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-test-1, groupId=test] Successfully synced group in generation Generation{generationId=2, memberId='consumer-test-1-c5924ab5-afc8-4720-a5d7-f8107ace3aad', protocol='range'}
2021-12-17 11:12:27.022 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-test-1, groupId=test] Notifying assignor about the new Assignment(partitions=[test-0])
2021-12-17 11:12:27.025 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-test-1, groupId=test] Adding newly assigned partitions: test-0
2021-12-17 11:12:27.029 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-test-1, groupId=test] Found no committed offset for partition test-0
2021-12-17 11:12:27.034 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-test-1, groupId=test] Found no committed offset for partition test-0
2021-12-17 11:12:27.040 INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.SubscriptionState : [Consumer clientId=consumer-test-1, groupId=test] Resetting offset for partition test-0 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka:9092 (id: 11 rack: null)], epoch=0}}.
2021-12-17 11:12:27.045 INFO 29067 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : test: partitions assigned: [test-0]
Solution
I’ve raised an issue to Spring kafka folks and it results to be a bug. No configuration is needed, spring application should quit as expected. Reference: https://github.com/spring-projects/spring-kafka/issues/2055
As stated in the issue, fix is cherry picked to spring-kafka 2.7.x, 2.6.x, 2.5.x. I’ll upgrade to the first 2.7.x patch release which includes the fix.
Answered By - tommaso.normani
Answer Checked By - Terry (JavaFixing Volunteer)
