Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Huggingface complains about a forked process when using BobCat and the QuantumModel #218

Open
dimkart opened this issue Feb 20, 2025 · 0 comments
Labels
fix A non-critical bug

Comments

@dimkart
Copy link
Contributor

dimkart commented Feb 20, 2025

Using BobCat to parse sentences, any method call of the QuantumModel will lead to the following warning message:

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

This spams the command-line output and needs to be fixed.

A hacky fix would be to add these two lines to the code:

import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"
@dimkart dimkart added the fix A non-critical bug label Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix A non-critical bug
Projects
None yet
Development

No branches or pull requests

1 participant