Official supported Python bindings for llama.cpp + gpt4all
For those who don't know, llama.cpp is a port of Facebook's LLaMA model in pure C/C++:
- Without dependencies
- Apple silicon first-class citizen - optimized via ARM NEON
- AVX2 support for x86 architectures
- Mixed F16 / F32 precision
- 4-bit quantization support
- Runs on the CPU
- The easy way is to use the prebuilt wheels
pip install pyllamacppHowever, the compilation process of llama.cpp is taking into account the architecture of the target CPU,
so you might need to build it from source:
git clone --recursive https://github.com/nomic-ai/pyllamacpp && cd pyllamacpp
pip install .A simple Pythonic API is built on top of llama.cpp C/C++ functions. You can call it from Python as follows:
from pyllamacpp.model import Model
def new_text_callback(text: str):
print(text, end="", flush=True)
model = Model(ggml_model='./models/gpt4all-model.bin', n_ctx=512)
model.generate("Once upon a time, ", n_predict=55, new_text_callback=new_text_callback, n_threads=8)If you don't want to use the callback, you can get the results from the generate method once the inference is finished:
generated_text = model.generate("Once upon a time, ", n_predict=55)
print(generated_text)If you want to run the program in interactive mode you can add the grab_text_callback function and set interactive to True in the generate function. grab_text_callback should always return a string unless you wish to signal EOF in which case you should return None.
from pyllamacpp.model import Model
def new_text_callback(text: str):
print(text, end="", flush=True)
def grab_text_callback():
inpt = input()
# To signal EOF, return None
if inpt == "END":
return None
return inpt
model = Model(ggml_model='./models/gpt4all-model.bin', n_ctx=512)
# prompt from https://github.com/ggerganov/llama.cpp/blob/master/prompts/chat-with-bob.txt
prompt = """
Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision. To do this, Bob uses a database of information collected from many different sources, including books, journals, online articles, and more.
User: Hello, Bob.
Bob: Hello. How may I help you today?
User: Please tell me the largest city in Europe.
Bob: Sure. The largest city in Europe is Moscow, the capital of Russia.
User:"""
model.generate(prompt, n_predict=256, new_text_callback=new_text_callback, grab_text_callback=grab_text_callback, interactive=True, repeat_penalty=1.0, antiprompt=["User:"])- You can pass any
llama contextparameter as a keyword argument to theModelclass - You can pass any
gptparameter as a keyword argument to thegenerartemethod - You can always refer to the short documentation for more details.
Download a GPT4All model from https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/.
The easiest approach is download a file whose name ends in ggml.bin--older model versions require conversion.
If you have an older model downloaded that you want to convert, in your terminal run:
pyllamacpp-convert-gpt4all path/to/gpt4all_model.bin path/to/llama_tokenizer path/to/gpt4all-converted.bin- Where to find the llama tokenizer? #5
If you find any bug, please open an issue.
If you have any feedback, or you want to share how you are using this project, feel free to use the Discussions and open a new topic.
This project is licensed under the same license as llama.cpp (MIT License).