Issue
ItemReader is reading from ever growing existing table everytime the job runs. I am looking for option within Spring batch to query only the new records every time the schedule job runs.
If I have 50000 records read, next schedule should start from 50001.
My thought is to persist the id of the last record read by ItemReader ( last of the whole reader output and not the last of each chunk ) in DB and use in subsequent job schedule. I will return the data sorted by id from main table.
How do I know the last record in the writer? Any ideas.
Solution
I would make it explicit by passing the ID range of the records (ie. fromId and toId) that are required to be processed as the job parameters when running a batch job. Then in the ItemReader, you can rely on this ID range to return the data to process.
And somehow persist the the latest ID that is already processed to the DB (e.g. by using JobExecutionListener when a job finished) . When the next scheduled job triggers, find out the next ID that is not processed and then start another job instance with this next ID as the parameter.
Answered By - Ken Chan
Answer Checked By - Terry (JavaFixing Volunteer)
