Why there is no need special token for chatglm3 when counting the tokens?

[condy0919]

LongBench/pred.py

Lines 58 to 59 in 26505c8

	if "chatglm3" in model_name:
	tokenized_prompt = tokenizer(prompt, truncation=False, return_tensors="pt", add_special_tokens=False).input_ids[0]

[bys0318]

Hi, you can change it to add_special_tokens=True, but I guess a few special tokens do not matter much compared to a context window of 32k tokens.

[condy0919]

I guess a few special tokens do not matter much compared to a context window of 32k tokens.

Agreed.