Issue
I've created: https://jira.spring.io/browse/BATCH-2778
I am developing Spring Batch + Redis (Spring Data Redis) example. In this example, I'm reading student.csv file and storing all the data as is in Redis DB. I wanted to used dateOfBirth as Date and I am sure that I need to do some date logic conversion to store value Date in Redis.
As per my analysis, it looks like I wont be able to use @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd"), because I am not dealing with JSON at all. I think I should be using the Converter etc to convert the Byte[] to Date and Date to Byte[] etc.
Infact I tried these options, but doesn't working. Any quick help?
Note: I don't want to save dateOfBirth as String value.
org.springframework.batch.item.file.FlatFileParseException: Parsing error at line: 2 in resource=[URL [file:c:/Videos/student.csv]], input=[1,John,Doe,05-12-1988 12:34:45]
at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFileItemReader.java:184) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.read(AbstractItemCountingItemStreamItemReader.java:89) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader$$FastClassBySpringCGLIB$$ebb633d0.invoke(<generated>) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.batch.item.file.FlatFileItemReader$$EnhancerBySpringCGLIB$$87932856.read(<generated>) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider.doRead(SimpleChunkProvider.java:91) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider.read(SimpleChunkProvider.java:157) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider$1.doInIteration(SimpleChunkProvider.java:116) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider.provide(SimpleChunkProvider.java:110) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:69) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:272) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:394) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:308) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:141) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:134) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_162]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_162]
at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_162]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE]
at com.sun.proxy.$Proxy69.run(Unknown Source) [na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:792) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:776) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1242) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1230) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE]
at com.myexample.DateBatchPocApplication.main(DateBatchPocApplication.java:12) [classes/:na]
Caused by: org.springframework.validation.BindException: org.springframework.validation.BeanPropertyBindingResult: 1 errors
Field error in object 'target' on field 'dateOfBirth': rejected value [05-12-1988 12:34:45]; codes [typeMismatch.target.dateOfBirth,typeMismatch.dateOfBirth,typeMismatch.java.util.Date,typeMismatch]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [target.dateOfBirth,dateOfBirth]; arguments []; default message [dateOfBirth]]; default message [Failed to convert property value of type 'java.lang.String' to required type 'java.util.Date' for property 'dateOfBirth'; nested exception is java.lang.IllegalStateException: Cannot convert value of type 'java.lang.String' to required type 'java.util.Date' for property 'dateOfBirth': no matching editors or conversion strategy found]
at org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper.mapFieldSet(BeanWrapperFieldSetMapper.java:200) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.item.file.mapping.DefaultLineMapper.mapLine(DefaultLineMapper.java:43) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFileItemReader.java:180) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE]
... 56 common frames omitted
StudentJob.java
@Configuration
public class StudentJob {
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Autowired
private FlatFileItemReader<Student> studentReader;
@Autowired
private StudentWritter studentWritter;
@Bean
public Job readstudentCSVFileJob() {
return jobBuilderFactory.get("readstudentCSVFileJob").incrementer(new RunIdIncrementer())
.start(countryCurrStepOne()).build();
}
@Bean
public Step countryCurrStepOne() {
return stepBuilderFactory.get("studentStepOne").<Student, Student>chunk(5).reader(studentReader)
.writer(studentWritter).build();
}
}
StudentBatchConfig.java
@Slf4j
@Configuration
public class StudentBatchConfig {
@Bean(destroyMethod="")
@StepScope
public FlatFileItemReader<Student> studentReader(@Value("${input.student.path}") Resource resource) throws IOException {
log.debug("Resource Path : "+resource.getFile());
FlatFileItemReader<Student> itemReader = new FlatFileItemReader<>();
itemReader.setName("STUDENT_READER");
itemReader.setResource(resource);
itemReader.setLineMapper(studentlineMapper());
itemReader.setLinesToSkip(1);
return itemReader;
}
@Bean
public LineMapper<Student> studentlineMapper() {
// Delimited Line Tokenizer
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setNames("id", "firstName", "lastName", "dateOfBirth");
lineTokenizer.setIncludedFields(new int[] {0,1,2,3});
// Bean Wrapper Field SetMapper
BeanWrapperFieldSetMapper<Student> fieldSetMapper = new BeanWrapperFieldSetMapper<>();
fieldSetMapper.setTargetType(Student.class);
DefaultLineMapper<Student> lineMapper = new DefaultLineMapper<>();
lineMapper.setLineTokenizer(lineTokenizer);
lineMapper.setFieldSetMapper(fieldSetMapper);
return lineMapper;
}
@Bean
public StudentWritter studentWritter() {
return new StudentWritter();
}
}
StudentWritter.java
public class StudentWritter implements ItemWriter<Student>{
@Autowired
private StudentRepository studentRepository;
@Override
public void write(List<? extends Student> students) throws Exception {
studentRepository.saveAll(students);
}
}
EDIT-1:
@Slf4j
@Configuration
public class StudentBatchConfig {
private static final SimpleDateFormat format = new SimpleDateFormat("dd-MM-yyyy HH:mm");
@Bean(destroyMethod="")
@StepScope
public FlatFileItemReader<Student> countryReader(@Value("${input.student.path}") Resource resource) throws IOException {
log.debug("Resource Path : "+resource.getFile());
FlatFileItemReader<Student> itemReader = new FlatFileItemReader<>();
itemReader.setName("STUDENT_READER");
itemReader.setResource(resource);
itemReader.setLineMapper(studentLineMapper());
itemReader.setLinesToSkip(1);
return itemReader;
}
@SuppressWarnings("rawtypes")
@Bean
public LineMapper<Student> studentLineMapper() {
DefaultLineMapper<Student> lineMapper = new DefaultLineMapper<>();
// Delimited Line Tokenizer
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setNames("id", "firstName", "lastName", "dateOfBirth");
lineTokenizer.setIncludedFields(new int[] {0,1,2,3});
// Date parsing logic has been added
CustomDateEditor customDateEditor = new CustomDateEditor(format, false);
HashMap<Class, PropertyEditor> customEditors = new HashMap<>();
customEditors.put(Date.class, customDateEditor);
// Bean Wrapper Field SetMapper
BeanWrapperFieldSetMapper<Student> fieldSetMapper = new BeanWrapperFieldSetMapper<>();
fieldSetMapper.setTargetType(Student.class);
fieldSetMapper.setCustomEditors(customEditors);
lineMapper.setLineTokenizer(lineTokenizer);
lineMapper.setFieldSetMapper(fieldSetMapper);
return lineMapper;
}
@Bean
public StudentWritter studentWritter() {
return new StudentWritter();
}
}
Student.java
@Builder
@Data
@AllArgsConstructor
@NoArgsConstructor
@RedisHash("student")
public class Student {
@Id @Indexed
private String id;
private String firstName;
private String lastName;
private Date dateOfBirth;
}
But dates are not storing correctly.. seem like formatting issue..
NOTE: If I open this file in CSV editor of Microsoft it shows me dd-MM-yyyy HH:mm
But If I edit this file in Notepad++, then it shows
id,firstName,lastName,dateOfBirth
1,John,Doe,25-11-2018 14:48:10
2,Sameer,Kumbhare,25-11-2018 14:48:10
Solution
Answrer posted here: https://jira.spring.io/browse/BATCH-2778
or
I used something like below. Looks like Redis doesn't worked well with the Java7 Date API, but it works well with the Java 8 API. But there is no out of the box support for the same.
public class StudentFieldSetMapper implements FieldSetMapper<Student>{
@Override
public Student mapFieldSet(FieldSet fieldSet) throws BindException {
return Student.builder()
.id(fieldSet.readString("id"))
.firstName(fieldSet.readString("firstName"))
.lastName(fieldSet.readString("lastName"))
.dateOfBirth(fieldSet.readDate("dateOfBirth", "dd-MM-yyyy HH:mm:ss")).build();
}
}
Answered By - PAA
Answer Checked By - Candace Johnson (JavaFixing Volunteer)
