-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Whether have a batch inference file? #8
Comments
Hi, if you want to use the SpikeGPT for batch inferencing, you can use the RWKV_GPT (in model.py) as the backbone instead of using RWKV_RNN, which provides fully support to the batch inference. For RWKV_RNN, since we expect it can process the variant sequence length, we do not support batch inference. |
Thx, I'll give it a try |
When I running model.py, I got this exception: |
Hi, this actually located in this github repo, you can check this link for what file you need, the 'cuda' folder should in the same path with your script. |
Thx for ur reply! It's ok when I run it on ubuntu. |
I wonder that whether there have a batch inference code?
It seems like I can only input one context for one time.
That would be nice if u can provided a file like batch_run.py or something else. Thx!
The text was updated successfully, but these errors were encountered: