Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Whether have a batch inference file? #8

Open
nanjunye opened this issue Nov 22, 2023 · 5 comments
Open

Whether have a batch inference file? #8

nanjunye opened this issue Nov 22, 2023 · 5 comments

Comments

@nanjunye
Copy link

I wonder that whether there have a batch inference code?
It seems like I can only input one context for one time.
That would be nice if u can provided a file like batch_run.py or something else. Thx!

@ridgerchu
Copy link
Owner

Hi, if you want to use the SpikeGPT for batch inferencing, you can use the RWKV_GPT (in model.py) as the backbone instead of using RWKV_RNN, which provides fully support to the batch inference. For RWKV_RNN, since we expect it can process the variant sequence length, we do not support batch inference.

@nanjunye
Copy link
Author

Thx, I'll give it a try

@nanjunye
Copy link
Author

nanjunye commented Nov 23, 2023

Hi, if you want to use the SpikeGPT for batch inferencing, you can use the RWKV_GPT (in model.py) as the backbone instead of using RWKV_RNN, which provides fully support to the batch inference. For RWKV_RNN, since we expect it can process the variant sequence length, we do not support batch inference.

When I running model.py, I got this exception:
Exception has occurred: ImportError
DLL load failed while importing wkv: 找不到指定的模块。
File "D:\GitHub\S_GPT\src\model.py", line 56, in
wkv_cuda = load(name="wkv", sources=["cuda/wkv_op.cpp", "cuda/wkv_cuda.cu"],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: DLL load failed while importing wkv: 找不到指定的模块。
It seems that it cannot find the module wkv, but I don't know where is wkv? pip install rwkv?

@ridgerchu
Copy link
Owner

Hi, this actually located in this github repo, you can check this link for what file you need, the 'cuda' folder should in the same path with your script.

@nanjunye
Copy link
Author

Hi, this actually located in this github repo, you can check this link for what file you need, the 'cuda' folder should in the same path with your script.

Thx for ur reply! It's ok when I run it on ubuntu.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants