Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce a new scrolling-based ItemReader #4484

Open
scordio opened this issue Nov 10, 2023 · 1 comment
Open

Introduce a new scrolling-based ItemReader #4484

scordio opened this issue Nov 10, 2023 · 1 comment
Labels
status: waiting-for-triage Issues that we did not analyse yet type: feature

Comments

@scordio
Copy link
Contributor

scordio commented Nov 10, 2023

Spring Data Commons 3.1 introduced the concept of scrolling (spring-projects/spring-data-commons#2787) and I'm particularly interested in scrolling using keyset-filtering, which provides performance benefits with large datasets.

It would be nice to have an item reader compatible with this concept.

As far as I can judge, RepositoryItemReader is highly coupled with PagingAndSortingRepository, therefore an extension doesn't seem to be the right direction as scrolling is supposed to work with any Repository.

Therefore, my proposal would be to introduce a new ItemReader, maybe a ScrollingRepositoryItemReader?

Happy to work on a PR in case the idea is accepted.

My concrete use case: in several Batch & JPA-based projects that are still on Spring Boot 2.7, scrolling based on keyset-filtering is currently achieved via a custom AbstractPaginatedItemReader with an underlying Spring Data JPA repository using QueryDSL predicates. After upgrading to Spring Boot 3, the QueryDSL predicates will be replaced by the Spring Data scrolling API. I would also be very happy to throw away the custom item reader if Spring Batch offered an idiomatic way to achieve the same result.

@scordio scordio added status: waiting-for-triage Issues that we did not analyse yet type: feature labels Nov 10, 2023
@scordio
Copy link
Contributor Author

scordio commented Dec 3, 2024

I'm reconsidering my previous suggestion to add a new ScrollingRepositoryItemReader.

Maybe there is a way to generalize RepositoryItemReader and make it work with the new scrolling patterns, without breaking existing code. I have a few new setters in mind, paired with an overall check in afterPropertiesSet to verify that the reader is configured for either the current behavior or the scrolling one.

I'll look into it and try to come up with a proposal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: waiting-for-triage Issues that we did not analyse yet type: feature
Projects
None yet
Development

No branches or pull requests

1 participant