Include limit of items read in JmsItemReader [BATCH-1752]

[spring-projects-issues]

Paulo Pimenta opened BATCH-1752 and commented

I developed a batch that needs to read objects from a JMS Queue and write them in a File.

The batch runs every 1 hour and should reads at most 500.000 records by execution and write them in the file.

I had to extend JmsItemReader, create limit and counter attributes and change the read method to read at most the limit attribute.

I would like to suggest to change JmsItemReader to have this feature.

---

Affects: 2.1.7

Attachments:

• JmsLimitedItemReader.java (1.12 kB)

[spring-projects-issues]

Lucas Ward commented

Its slightly trickier than that. Adding some type of limit would be making the reader stateful. As it stands now, its stateless and as a result is one of the few readers that can easily be used in a multi-threaded step. So, adding potential stateful scenarios may not be wise. However, it might make sense to create a new reader extending the old called StatefulJmsItemReader, with a simple interface allowing for various implementations to decide when the reading can be considered 'done'. It would also likely need to store this in the ExecutionContext, because otherwise it wouldn't handle restart scenarios correctly, and once you've read and removed the items from the queue, you can't exactly go back. I'm curious what Dave thinks as well, since he has had more experience in the multi-threaded area with Spring Batch, and wrote the original implementation.

[spring-projects-issues]

Lucas Ward commented

I just took a look at your attached class. My concerns from above about the ExecutionContext are valid for you implementation as well. Because you're not storing the readCounter on itemStream#update(), the counter would start over again at 0 if the job failed and had to be restarted.

[spring-projects-issues]

Dave Syer commented

I think we can add a counter. As long as it is only incemented (and is implemented in a thread safe wrapper) I think the net result will be that a restart will finish the job so that the total number of items is at most the max limit.