[FIXED] Why are the data sources interfering in Spring Batch when using a RepositoryItemReader?

Issue

I am trying to migrate some data between a Postgres database and MongoDB using Spring Batch. I have a very simple ItemReader, ItemProcessor, and ItemWriter configured, and it everything works as intended. However, if I switch to a RepositoryItemReader, I'm getting the following error:

java.lang.IllegalStateException: Already value [org.springframework.jdbc.datasource.ConnectionHolder@684430c1] for key [HikariDataSource (HikariPool-1)] bound to thread

If I understand correctly, there is something wrong with the EntityManager or TransactionManager, but I cannot figure out what, and why it's working with a simple ItemReader that doesn't work with a repository, but it uses the same data source.

I would be very grateful for any help.

Here is my source db configuration:

package com.example.batch.primary;

@Configuration
@EnableTransactionManagement
@EnableJpaRepositories(
    entityManagerFactoryRef = "primaryEntityManagerFactory",
    transactionManagerRef = "primaryTransactionManager",
    basePackages = {"com.example.batch.primary"}
    )
public class PrimaryDBConfig {

@Bean(name = "primaryDataSource")
@Primary
public DataSource primaryDatasource(){

    DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create()
                     .driverClassName("org.postgresql.Driver")
                     .url("jdbc:postgresql://localhost:5432/postgres")
                     .username("test")
                     .password("test");
    return dataSourceBuilder.build();
}


@Bean(name = "primaryEntityManagerFactory")
@Primary
public LocalContainerEntityManagerFactoryBean primaryEntityManagerFactory(EntityManagerFactoryBuilder builder,
                                                                          @Qualifier("primaryDataSource")
                                                                          DataSource primaryDataSource){
    return builder.dataSource(primaryDataSource)
                  .packages("com.example.batch.primary")
                  .build();
}

@Bean(name = "primaryTransactionManager")
public PlatformTransactionManager primaryTransactionManager(
        @Qualifier("primaryEntityManagerFactory") EntityManagerFactory primaryEntityManagerFactory)
{
    return new JpaTransactionManager(primaryEntityManagerFactory);
}
}

Here is the configuration of MongoDB:

package com.example.batch.secondary;

@EnableMongoRepositories(basePackages = "com.example.batch.secondary")
@Configuration
public class MongoDBConfig {

@Bean
public MongoClient mongo() {
    ConnectionString connectionString = new ConnectionString("mongodb+srv://mongoadmin:blablabla.mongodb.net/?retryWrites=true&w=majority");
    MongoClientSettings mongoClientSettings = MongoClientSettings.builder()
            .applyConnectionString(connectionString)
            .build();

    return MongoClients.create(mongoClientSettings);
}

@Bean
public MongoTemplate mongoTemplate() throws Exception {
    return new MongoTemplate(mongo(), "test");
}
}

Here is the RepositoryItemReader:

package com.example.batch.stepcomponents;

@Component
public class RepositoryReader extends RepositoryItemReader<Partner> {

public RepositoryReader(@Autowired PartnerRepository partnerRepository){
    setRepository(partnerRepository);
    setPageSize(1);
    setSort(Map.of("id", Sort.Direction.ASC));
    setMethodName("findAll");
}
}

Batch Config:

@Configuration
@EnableBatchProcessing
public class BatchConfig {

@Autowired
public JobBuilderFactory jobBuilderFactory;

@Autowired
public StepBuilderFactory stepBuilderFactory;

@Autowired
RepositoryReader repositoryReader;

@Autowired
CustomWriter customWriter;

@Autowired
CustomProcessor customProcessor;

@Bean
public Job createJob() {
    return jobBuilderFactory.get("MyJob")
            .incrementer(new RunIdIncrementer())
            .flow(createStep())
            .end()
            .build();
}

@Bean
public Step createStep() {

    return stepBuilderFactory.get("MyStep")
            .<Partner, Student> chunk(1)
            .reader(repositoryReader)
            .processor(customProcessor)
            .writer(customWriter)
            .build();
}
}

Solution

So I tried taking out the EntityManagerFactory and the TransactionManager, and now it works. I guess they are already initialized automatically when starting up the server..

Yes, by default, if you provide a DataSource bean, Spring Batch will use a DataSourceTransactionManager, not the JPA one as you expect. This is explained in the Javadoc of EnableBatchProcessing:

The transaction manager provided by this annotation will be of type:
 * ResourcelessTransactionManager if no DataSource is provided within the context
 * DataSourceTransactionManager if a DataSource is provided within the context

In order to use the JPA transaction manager, you need to configure a custom a BatchConfigurer and override getTransactionManager, something like:

@Bean
public BatchConfigurer batchConfigurer(DataSource dataSource, EntityManagerFactory entityManagerFactory) {
    return new DefaultBatchConfigurer(dataSource) {
        @Override
        public PlatformTransactionManager getTransactionManager() {
            return new JpaTransactionManager(entityManagerFactory);
        }
    };
}

Note this will not be required anymore starting from v5, see:

• Revisit the configuration of infrastructure beans with @EnableBatchProcessing
• Spring Batch 5.0.0-M6 and 4.3.7 are out!

You can also set the JPA transaction manager on your step:

@Bean
public Step createStep(JpaTransactionManager jpaTransactionManager) {

    return stepBuilderFactory.get("MyStep")
            .<Partner, Student> chunk(1)
            .reader(repositoryReader)
            .processor(customProcessor)
            .writer(customWriter)
            .transactionManager(jpaTransactionManager)
            .build();
}

Answered By - Mahmoud Ben Hassine
Answer Checked By - Mildred Charles (JavaFixing Admin)