[FIXED] Spring Boot 2.7.2 with Spring Kafka fails during initialisation with Exception: The 'ProducerFactory' must support transactions`

Issue

I'm using Spring Boot v2.7.2 and the latest version of Spring Kafka provided by spring-boot-dependencies:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

I want the app to load all configuration from file hence I created the beans with this bare minimum configuration:

    public class KakfaConfig {

    @Bean
    public ProducerFactory<Integer, FileUploadEvent> producerFactory() {
        return new DefaultKafkaProducerFactory<>(Collections.emptyMap());
    }

    @Bean
    public KafkaTemplate<Integer, FileUploadEvent> kafkaTemplate() {
        return new KafkaTemplate<Integer, anEvent>(producerFactory());
    }

}

It works and loads the configuration from the application.yaml below as expected.

spring:
  application:
    name: my-app
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      client-id: ${spring.application.name}
  #    transaction-id-prefix: "tx-"
    template:
      default-topic: my-topic

However, if I uncomment the transaction-id-prefix line, the application fails to start with the exception java.lang.IllegalArgumentException: The 'ProducerFactory' must support transactions
The documentation in here reads

If you provide a custom producer factory, it must support transactions. See ProducerFactory.transactionCapable().

The only way I managed to make it work is removing the transaction prefix from the application.yaml and configure it in the code as per below:

@Bean
public ProducerFactory<Integer, FileUploadEvent> fileUploadProducerFactory() {
    var pf = new DefaultKafkaProducerFactory<Integer, FileUploadEvent>(Collections.emptyMap());
    pf.setTransactionIdPrefix("tx-");
    return pf;
}

Any thoughts on how I can configure everything using the application properties file? Is this a bug?

Solution

The only solution atm is really setting the transaction-prefix-id in the code whilst creating the ProducerFactory, despite it's already been defined in the application.yaml.

The Spring Boot team replied as per below:

The intent is that transactions should be used and that the ProducerFactory should support them. The transaction-id-prefix property can be set and this results in the auto-configuration of the kafkaTransactionManager bean. However, if you define your own ProducerFactory (to constrain the types, for example) there's no built-in way to have the transaction-id-prefix applied to that ProducerFactory.

It's a fundamental principle of auto-configuration that it backs off when a user defines a bean of their own. If we post-processed the user's bean to change its configuration, it would no longer be possible for your code to take complete control of how things are configured. Unfortunately, this flexibility does sometimes require you to write a little bit more code. This is one such time.

If we want to keep the prefix as a property in the application.yaml file, we can inject it to avoid config duplication:

@Value("${spring.kafka.producer.transaction-id-prefix}")
private String transactionPrefix;

@Bean
public ProducerFactory<Integer, FileUploadEvent> fileUploadProducerFactory() {
    var pf = new DefaultKafkaProducerFactory<Integer, FileUploadEvent>(Collections.emptyMap());
    pf.setTransactionIdPrefix(transactionIdPrefix);
    return pf;
}

Answered By - dbaltor
Answer Checked By - Candace Johnson (JavaFixing Volunteer)