Skip to content

Issues: openai/tiktoken

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
0 Open 97 Closed
0 Open 97 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Missing git tag
#382 by frenzymadness was closed Feb 14, 2025
1
Add o3-mini support
#374 by Nicba1010 was closed Feb 14, 2025
11
How to use in full offline env?
#369 by momomobinx was closed Jan 15, 2025
1
tiktoken.encoding_for_model fails for o1
#367 by Somerandomguy10111 was closed Jan 28, 2025
Tokenizer for o1
#363 by texasroh was closed Feb 14, 2025
4
ValueError: not enough values to unpack (expected 2, got 1).
#353 by Itime-ren was closed Feb 14, 2025
7
Python 3.11 wheel aarch64 missing for tiktoken 0.8
#347 by hauntsaninja was closed Oct 4, 2024
2
Does tiktoken count only input tokens or output tokens as well?
#344 by GildeshAbhay was closed Oct 3, 2024
1
Is there a new tokenizer for o1 models?
#337 by jiadingfang was closed Oct 3, 2024
9
chatgpt-4o-latest is not yet added
#335 by jvlinsta was closed Oct 3, 2024
3
Facing erros in importing the o200k_base
#334 by JaynouOliver was closed Oct 3, 2024
9
Leveraging DP for bpe_encode function
#333 by lordgavy01 was closed Aug 22, 2024
ValueError: not enough values to unpack (expected 2, got 1) when tiktoken.get_encoding("cl100k_base")
#330 by hzh12345678 was closed Oct 6, 2024
2
RecursionError: maximum recursion depth exceeded while calling a Python object
#323 by Hudrolax was closed Jul 19, 2024
5
Support for GPT 4o
#305 by jcrupi was closed Jun 10, 2024
1
Unknown encoding gpt2
#301 by aryagxr was closed May 27, 2024
1
o200k_base pretokenizer - regex error?
#298 by AmitMY was closed May 18, 2024
2
gpt-4o tokenizer
#295 by nxfi777 was closed May 13, 2024
1
Combining marks and indic vowel marks within words are being split breaking all indic languages and most languages except English and CJKs
#292 by ajaykg was closed May 5, 2024
4
Custom tokenizer fails to encode despite characters being in mergeable_ranks
#289 by afang-story was closed Oct 3, 2024
3
Using offline: .tiktoken file gets deleted automatically on Linux
#279 by nkilm was closed Aug 22, 2024
4
Encode an empty string gives empty tokens
#276 by flexwang was closed Apr 12, 2024
2
Python memory usage
#242 by logan-markewich was closed Jan 30, 2024
6
new method to truncate after N-tokens
#236 by aleks-sch was closed Jan 17, 2024
2
Is the model link invalid？
#233 by igorwang was closed Jan 17, 2024
ProTip! Updated in the last three days: updated:>2025-02-12.