Number of tokens (757) exceeded maximum context length (512).

[datacrud8]

hi, trying to build this app in local, and used same model llama-2-7b-chat.ggmlv3.q8_0.bin
when run the app UI showing some random message same like you showed but checking in console getting this below message:

Number of tokens (755) exceeded maximum context length (512).
Number of tokens (756) exceeded maximum context length (512).
Number of tokens (757) exceeded maximum context length (512).

so increased max_new_tokens=2048, and increased n_ctx and added truncate=True , non of them are fixing this issue.
Changed the model as well. still same issue.

do you know any solution for this issue?

[ctxwing]

i got the just same as @datacrud8 .

did any one got solved ? thanks in advance.

$ chainlit run main.py -w
2023-10-25 19:38:13 - Loaded .env file
2023-10-25 19:38:22 - Your app is available at http://localhost:8000
2023-10-25 19:38:51 - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2
2023-10-25 19:38:54 - Anonymized telemetry enabled. See https://docs.trychroma.com/telemetry for more information.
2023-10-25 19:39:06 - Load pretrained SentenceTransformer: sentence-transformers/all-MiniLM-L6-v2
2023-10-25 19:39:07 - Anonymized telemetry enabled. See https://docs.trychroma.com/telemetry for more information.
Batches: 100%|██████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00, 2.15s/it]
2023-10-25 19:39:20 - 4 changes detected
Batches: 100%|██████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 6.11it/s]
2023-10-25 19:41:53 - Number of tokens (513) exceeded maximum context length (512).
2023-10-25 19:41:53 - Number of tokens (514) exceeded maximum context length (512).
2023-10-25 19:41:54 - Number of tokens (515) exceeded maximum context length (512).
2023-10-25 19:41:54 - Number of tokens (516) exceeded maximum context length (512).
2023-10-25 19:41:55 - Number of tokens (517) exceeded maximum context length (512).
2023-10-25 19:41:55 - Number of tokens (518) exceeded maximum context length (512).

[sudarshan-koirala]

hello, can you try a different embeddings model, for example, hkunlp/instructor-large in the ingest.py file.

[ctxwing]

@sudarshan-koirala At first, thanks for the answer to my question.
i changed this
model_name="sentence-transformers/all-MiniLM-L6-v2" ... result is case [A]

into below lists including yours recommend, from by refering (https://www.sbert.net/docs/pretrained_models.html#multi-lingual-models ) .

• hkunlp/instructor-large ... result is case [B]
• sentence-transformers/multi-qa-MiniLM-L6-cos-v1 ... result is case [A]

but got no luck ...
did i miss something ?

   huggingface_embeddings = HuggingFaceEmbeddings(
       model_name="sentence-transformers/all-MiniLM-L6-v2", #<-- [A] this cause exceeded maximum context length (512).
       model_kwargs={"device": "cpu"},
       )

    huggingface_embeddings = HuggingFaceEmbeddings(
        model_name="hkunlp/instructor-large", #<-- [B] changed to this , throw error below
        model_kwargs={"device": "cpu"},
        )

$ time python ingest.py
Downloading (…)c7233/.gitattributes: 100%|████████████████████████████████████████████████| 1.48k/1.48k [00:00<00:00, 3.60MB/s]
Downloading (…)_Pooling/config.json: 100%|█████████████████████████████████████████████████████| 270/270 [00:00<00:00, 716kB/s]
Downloading (…)/2_Dense/config.json: 100%|█████████████████████████████████████████████████████| 116/116 [00:00<00:00, 291kB/s]
Downloading pytorch_model.bin: 100%|██████████████████████████████████████████████████████| 3.15M/3.15M [00:00<00:00, 11.1MB/s]
Downloading (…)9fb15c7233/README.md: 100%|█████████████████████████████████████████████████| 66.3k/66.3k [00:00<00:00, 338kB/s]
Downloading (…)b15c7233/config.json: 100%|████████████████████████████████████████████████| 1.53k/1.53k [00:00<00:00, 4.31MB/s]
Downloading (…)ce_transformers.json: 100%|█████████████████████████████████████████████████████| 122/122 [00:00<00:00, 358kB/s]
Downloading pytorch_model.bin: 100%|██████████████████████████████████████████████████████| 1.34G/1.34G [01:56<00:00, 11.5MB/s]
Downloading (…)nce_bert_config.json: 100%|███████████████████████████████████████████████████| 53.0/53.0 [00:00<00:00, 157kB/s]
Downloading (…)cial_tokens_map.json: 100%|████████████████████████████████████████████████| 2.20k/2.20k [00:00<00:00, 6.51MB/s]
Downloading spiece.model: 100%|█████████████████████████████████████████████████████████████| 792k/792k [00:00<00:00, 11.9MB/s]
Downloading (…)c7233/tokenizer.json: 100%|████████████████████████████████████████████████| 2.42M/2.42M [00:01<00:00, 2.36MB/s]
Downloading (…)okenizer_config.json: 100%|████████████████████████████████████████████████| 2.41k/2.41k [00:00<00:00, 7.06MB/s]
Downloading (…)15c7233/modules.json: 100%|████████████████████████████████████████████████████| 461/461 [00:00<00:00, 1.37MB/s]
Traceback (most recent call last):
File "/home/ctxwing/docker-ctx/lancer/basic/llama2-chat-with-documents/ingest.py", line 82, in
create_vector_database()
File "/home/ctxwing/docker-ctx/lancer/basic/llama2-chat-with-documents/ingest.py", line 59, in create_vector_database
huggingface_embeddings = HuggingFaceEmbeddings(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/ctxwing/anaconda3/envs/py311-chainlit/lib/python3.11/site-packages/langchain/embeddings/huggingface.py", line 66, in init
self.client = sentence_transformers.SentenceTransformer(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ctxwing/anaconda3/envs/py311-chainlit/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 95, in init
modules = self._load_sbert_model(model_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ctxwing/anaconda3/envs/py311-chainlit/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 840, in _load_sbert_model
module = module_class.load(os.path.join(model_path, module_config['path']))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ctxwing/anaconda3/envs/py311-chainlit/lib/python3.11/site-packages/sentence_transformers/models/Pooling.py", line 120, in load
return Pooling(**config)
^^^^^^^^^^^^^^^^^
TypeError: Pooling.init() got an unexpected keyword argument 'pooling_mode_weightedmean_tokens'

real 2m12.113s
user 0m14.215s
sys 0m9.627s

[sny-verma]

As per new updates, define like this:-

llm = CTransformers(
model=model_path,
model_type=model_type,
config={'max_new_tokens': 1024,
'temperature': 0.7,
'context_length': 4096}
)