Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

max_workers limited to 61 #201

Closed
rileyg98 opened this issue Aug 27, 2022 · 2 comments
Closed

max_workers limited to 61 #201

rileyg98 opened this issue Aug 27, 2022 · 2 comments

Comments

@rileyg98
Copy link

Latest pypi version - I've just tried to use Swifter for the first time and it appears there's a max_workers limit of 61. Given swifters default of 2*cpu_core this is lower than a conceivable number - a 32-core Threadripper 3970x will go above this limit.

Either setting a maximum of 61 when calculating the default, or expanding the max_workers limit would resolve this issue.

@rileyg98
Copy link
Author

It would appear that this issue is related to cpython on Windows.

@jmcarpenter2
Copy link
Owner

Swifter defaults npartitions to 2*cpu_core, but you can control that parameter with df.swifter.set_npartitions or with the set_defaults functionality

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants