Skip to content

Issues: openai/tiktoken

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
0 Open 98 Closed
0 Open 98 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

When I was inputting long text into a large model, that is, when the len of the text was 1024*1024, a StackOverflow error occurred.
#327 by YangQiangli was closed Feb 16, 2025 updated Feb 16, 2025
2
Add o3-mini support
#374 by Nicba1010 was closed Feb 14, 2025 updated Feb 14, 2025
11
Missing git tag
#382 by frenzymadness was closed Feb 14, 2025 updated Feb 14, 2025
1
Tokenizer for o1
#363 by texasroh was closed Feb 14, 2025 updated Feb 14, 2025
4
ValueError: not enough values to unpack (expected 2, got 1).
#353 by Itime-ren was closed Feb 14, 2025 updated Feb 14, 2025
7
ValueError: Unknown encoding cl100k_base
#80 by rembo1992 was closed Mar 27, 2023 updated Feb 12, 2025
6
tiktoken.encoding_for_model fails for o1
#367 by Somerandomguy10111 was closed Jan 28, 2025 updated Jan 28, 2025
How to use in full offline env?
#369 by momomobinx was closed Jan 15, 2025 updated Jan 15, 2025
1
Exporting BPE file to vocab/merges files used by Huggingface tokenizers
#60 by colinclement was closed Mar 31, 2023 updated Dec 6, 2024
16
Connection reset by peer when the server get bpe file
#211 by leantli was closed Nov 7, 2023 updated Nov 27, 2024
2
ValueError: not enough values to unpack (expected 2, got 1) when tiktoken.get_encoding("cl100k_base")
#330 by hzh12345678 was closed Oct 6, 2024 updated Oct 6, 2024
2
Thread Panic when decoding token id 100256 and others with cl100k_base tokenizer
#47 by minimaxir was closed Oct 6, 2024 updated Oct 6, 2024
3
Python 3.11 wheel aarch64 missing for tiktoken 0.8
#347 by hauntsaninja was closed Oct 4, 2024 updated Oct 4, 2024
2
Custom tokenizer fails to encode despite characters being in mergeable_ranks
#289 by afang-story was closed Oct 3, 2024 updated Oct 3, 2024
3
chatgpt-4o-latest is not yet added
#335 by jvlinsta was closed Oct 3, 2024 updated Oct 3, 2024
3
Is there a new tokenizer for o1 models?
#337 by jiadingfang was closed Oct 3, 2024 updated Oct 3, 2024
9
Plugins found: ['tiktoken_ext.openai_public']
#218 by voghoei was closed Oct 3, 2024 updated Oct 3, 2024
15
Adding a __version__ attribute
#190 by rasbt was closed Oct 3, 2024 updated Oct 3, 2024
1
5
Facing erros in importing the o200k_base
#334 by JaynouOliver was closed Oct 3, 2024 updated Oct 3, 2024
9
Does tiktoken count only input tokens or output tokens as well?
#344 by GildeshAbhay was closed Oct 3, 2024 updated Oct 3, 2024
1
RecursionError: maximum recursion depth exceeded while calling a Python object
#323 by Hudrolax was closed Jul 19, 2024 updated Aug 29, 2024
5
Using offline: .tiktoken file gets deleted automatically on Linux
#279 by nkilm was closed Aug 22, 2024 updated Aug 22, 2024
4
Leveraging DP for bpe_encode function
#333 by lordgavy01 was closed Aug 22, 2024 updated Aug 22, 2024
Can't install tiktoken in Python 3.12
#205 by pamelafox was closed Dec 3, 2023 updated Jul 18, 2024
7
Error: pyo3_runtime.PanicException assertion failed: encoder.len() == decoder.len()
#87 by osjason was closed Apr 5, 2023 updated Jul 5, 2024
4
ProTip! Updated in the last three days: updated:>2025-02-16.