Can i run on windows?

[talhaanwarch]

I tried to run on windows and i get this error

ERROR:    Traceback (most recent call last):
  File "C:\streamlit-llama\venv\lib\site-packages\starlette\routing.py", line 677, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "C:\streamlit-llama\venv\lib\site-packages\starlette\routing.py", line 566, in __aenter__
    await self._router.startup()
  File "C:\streamlit-llama\venv\lib\site-packages\starlette\routing.py", line 654, in startup
    await handler()
  File "C:\streamlit-llama\api.py", line 22, in startup_event
    app.state.model = load_model()
  File "C:\streamlit-llama\api.py", line 15, in load_model
    return AutoModelForCausalLM.from_pretrained(
  File "C:\streamlit-llama\venv\lib\site-packages\ctransformers\hub.py", line 168, in from_pretrained
    return LLM(
  File "C:\streamlit-llama\venv\lib\site-packages\ctransformers\llm.py", line 206, in __init__
    self._lib = load_library(lib, cuda=config.gpu_layers > 0)
  File "C:\streamlit-llama\venv\lib\site-packages\ctransformers\llm.py", line 102, in load_library
    lib = CDLL(path)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\ctypes\__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'C:\streamlit-llama\venv\Lib\site-packages\ctransformers\lib\avx2\ctransformers.dll' (or one of its dependencies). Try using the full path with constructor syntax.

ERROR:    Application startup failed. Exiting.

Windows 11

[bbartling]

Windows 10 and python 3.10 is working for me just fine.

[bbartling]

I am just experimenting with this on the fly without even a virtual environment and havent ran into any issues, here's a pip list on my end:

> py -3.10 -m  pip list
Package                      Version
---------------------------- -----------
absl-py                      1.4.0
aiohttp                      3.8.3
aiohttp-basicauth            1.0.0
aiosignal                    1.3.1
ale-py                       0.7.4
altair                       4.2.0
altgraph                     0.17.3
annoy                        1.17.3
anyio                        3.6.2
APScheduler                  3.9.1.post1
argcomplete                  3.1.1
argon2-cffi                  21.3.0
argon2-cffi-bindings         21.2.0
arrow                        1.2.3
arviz                        0.12.1
asciichartpy                 1.5.25
asttokens                    2.2.1
astunparse                   1.6.3
async-timeout                4.0.2
asyncio-mqtt                 0.16.1
attrs                        22.2.0
autopep8                     1.7.0
AutoROM                      0.6.1
AutoROM.accept-rom-license   0.6.1
BAC0                         21.12.3
backcall                     0.2.0
backoff                      2.2.1
bacpypes                     0.18.6
Beaker                       1.12.1
beautifulsoup4               4.12.2
black                        23.3.0
bleach                       6.0.0
blinker                      1.5
branca                       0.6.0
Brotli                       1.0.9
cachetools                   5.2.0
censys                       2.2.4
certifi                      2022.12.7
cffi                         1.15.1
cftime                       1.6.2
charset-normalizer           2.1.1
chroma-hnswlib               0.7.1
chromadb                     0.4.3
click                        8.1.3
click-plugins                1.1.1
cligj                        0.7.2
cloudpickle                  2.2.1
colorama                     0.4.5
coloredlogs                  15.0.1
comm                         0.1.3
commonmark                   0.9.1
contourpy                    1.0.6
cryptography                 39.0.0
cssselect2                   0.7.0
ctransformers                0.2.20
cvxpy                        1.3.1
cycler                       0.11.0
dataclasses-json             0.5.13
datasets                     2.14.1
debugpy                      1.6.7
decorator                    5.1.1
defusedxml                   0.7.1
deprecat                     2.1.1
dill                         0.3.7
diskcache                    5.6.1
docopt                       0.6.2
docx                         0.2.4
ecos                         2.0.12
eemeter                      3.1.1
energyplus-api-helpers       0.4
entrypoints                  0.4
et-xmlfile                   1.1.0
exceptiongroup               1.1.0
executing                    1.2.0
faiss-cpu                    1.7.4
fastapi                      0.95.2
fastjsonschema               2.16.3
fastprogress                 1.0.3
filelock                     3.12.0
Fiona                        1.9.4.post1
Flask                        2.2.2
Flask-Classful               0.14.2
Flask-Cors                   3.0.10
Flask-Pydantic               0.11.0
Flask-SQLAlchemy             3.0.2
flatbuffers                  23.1.21
folium                       0.14.0
fonttools                    4.38.0
fqdn                         1.5.1
frozenlist                   1.3.3
fsspec                       2023.6.0
future                       0.18.2
g36-ahu-faults               0.0.1
gast                         0.4.0
gensim                       4.3.1
geopandas                    0.13.2
gitdb                        4.0.10
GitPython                    3.1.29
google-auth                  2.16.0
google-auth-oauthlib         0.4.6
google-pasta                 0.2.0
greenlet                     2.0.1
grpcio                       1.51.1
gym                          0.21.0
h11                          0.14.0
h5netcdf                     1.1.0
h5py                         3.7.0
html5lib                     1.1
httptools                    0.6.0
huggingface-hub              0.16.4
humanfriendly                10.0
idna                         3.4
importlib-metadata           4.13.0
importlib-resources          5.12.0
iniconfig                    2.0.0
install                      1.3.5
InstructorEmbedding          1.0.1
ipykernel                    6.22.0
ipython                      8.13.2
ipython-genutils             0.2.0
ipywidgets                   8.0.6
isoduration                  20.11.0
itsdangerous                 2.1.2
jedi                         0.18.2
Jinja2                       3.1.2
joblib                       1.2.0
jsonpointer                  2.3
jsonschema                   4.17.3
jupyter                      1.0.0
jupyter_client               8.2.0
jupyter-console              6.6.3
jupyter_core                 5.3.0
jupyter-events               0.6.3
jupyter_server               2.5.0
jupyter_server_terminals     0.4.4
jupyterlab-pygments          0.2.2
jupyterlab-widgets           3.0.7
keras                        2.11.0
keyboard                     0.13.5
kiwisolver                   1.4.4
langchain                    0.0.247
langsmith                    0.0.15
Levenshtein                  0.21.0
libclang                     15.0.6.1
llama-cpp-python             0.1.77
lxml                         4.9.2
Markdown                     3.4.1
MarkupSafe                   2.1.2
marshmallow                  3.20.1
matplotlib                   3.6.2
matplotlib-inline            0.1.6
matplotlib2tikz              0.7.6
mistune                      2.0.5
modbus-tk                    1.1.2
monotonic                    1.6
mpmath                       1.3.0
multidict                    6.0.4
multiprocess                 0.70.15
mypy-extensions              1.0.0
nbclassic                    1.0.0
nbclient                     0.7.4
nbconvert                    7.3.1
nbformat                     5.8.0
nest-asyncio                 1.5.6
netCDF4                      1.6.3
nltk                         3.8.1
notebook                     6.5.4
notebook_shim                0.2.3
num2words                    0.5.12
numexpr                      2.8.4
numpy                        1.25.1
oauthlib                     3.2.2
onnxruntime                  1.15.1
openai                       0.27.8
openapi-schema-pydantic      1.2.4
opencv-python                4.6.0.66
openleadr                    0.5.28
openpyxl                     3.1.2
openvino                     2022.3.0
opt-einsum                   3.3.0
osqp                         0.6.2.post8
overrides                    7.3.1
packaging                    23.1
paho-mqtt                    1.6.1
pandas                       1.5.0
pandocfilters                1.5.0
parso                        0.8.3
pathspec                     0.11.1
patsy                        0.5.3
pdf2image                    1.16.3
pdfkit                       1.0.0
pdfminer.six                 20221105
pdfminer3k                   1.3.4
pdfplumber                   0.10.1
pefile                       2022.5.30
pickleshare                  0.7.5
Pillow                       9.3.0
pip                          23.2.1
platformdirs                 3.5.0
pluggy                       1.0.0
ply                          3.11
posthog                      3.0.1
prometheus-client            0.16.0
prompt-toolkit               3.0.38
protobuf                     4.23.4
psutil                       5.9.5
PuLP                         2.7.0
pulsar-client                3.2.0
pump                         0.1.2
pure-eval                    0.2.2
py-cpuinfo                   9.0.0
pyarrow                      10.0.1
pyasn1                       0.4.8
pyasn1-modules               0.2.8
pycausalimpact               0.1.1
pycocotools                  2.0.6
pycodestyle                  2.9.1
pycparser                    2.21
pydantic                     1.10.2
pydeck                       0.8.0
pydyf                        0.6.0
pygame                       2.4.0
pyglet                       1.5.27
Pygments                     2.13.0
pyinstaller                  5.6.2
pyinstaller-hooks-contrib    2022.13
pymc3                        3.11.5
pymodbus                     3.0.2
Pympler                      1.0.1
pyOpenSSL                    23.0.0
pyparsing                    3.0.9
PyPDF2                       3.0.1
pypdfium2                    4.18.0
pyphen                       0.14.0
PyPika                       0.48.9
pyproj                       3.6.0
PyQt5                        5.15.7
PyQt5-Qt5                    5.15.2
PyQt5-sip                    12.11.0
pyreadline3                  3.4.1
pyrsistent                   0.19.2
pyserial                     3.5
pysparklines                 1.4
pytesseract                  0.3.10
pytest                       7.2.1
python-dateutil              2.8.2
python-docx                  0.8.11
python-dotenv                1.0.0
python-json-logger           2.0.7
python-Levenshtein           0.21.0
python-statemachine          2.0.0
pytz                         2022.7.1
pytz-deprecation-shim        0.1.0.post0
pywin32                      306
pywin32-ctypes               0.2.0
pywinpty                     2.0.10
PyYAML                       6.0
pyzmq                        25.0.2
qdldl                        0.1.5.post3
qtconsole                    5.4.2
QtPy                         2.3.1
rapidfuzz                    3.0.0
regex                        2023.5.5
requests                     2.31.0
requests-oauthlib            1.3.1
rfc3339-validator            0.1.4
rfc3986-validator            0.1.1
rich                         12.6.0
rsa                          4.9
safetensors                  0.3.1
scapy                        2.4.5
scikit-learn                 1.2.0
scipy                        1.7.3
scs                          3.2.2
seaborn                      0.12.2
semver                       2.13.0
Send2Trash                   1.8.2
sentence-transformers        0.4.1.2
sentencepiece                0.1.99
setuptools                   66.1.1
shapely                      2.0.1
signxml                      2.10.1
sip                          6.7.4
six                          1.16.0
sklearn                      0.0.post1
slate3k                      0.5.3
smart-open                   6.3.0
smmap                        5.0.0
sniffio                      1.3.0
soupsieve                    2.4.1
SQLAlchemy                   1.4.45
stable-baselines3            1.8.0
stack-data                   0.6.2
starlette                    0.27.0
statsmodels                  0.14.0
streamlit                    1.25.0
swig                         4.1.1
sympy                        1.11.1
tenacity                     8.2.2
tensorboard                  2.11.2
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.1
tensorflow                   2.11.0
tensorflow-estimator         2.11.0
tensorflow-intel             2.11.0
tensorflow-io-gcs-filesystem 0.30.0
termcolor                    2.2.0
terminado                    0.17.1
Theano-PyMC                  1.1.2
threadpoolctl                3.1.0
tikzplotlib                  0.10.1
tinycss2                     1.2.1
tokenizers                   0.13.3
toml                         0.10.2
tomli                        2.0.1
toolz                        0.12.0
torch                        1.13.0
torchvision                  0.14.0
tornado                      6.2
tqdm                         4.65.0
traitlets                    5.9.0
transformers                 4.31.0
typing_extensions            4.7.1
typing-inspect               0.9.0
tzdata                       2022.7
tzlocal                      4.2
UliEngineering               0.3.10
uri-template                 1.2.0
urllib3                      1.26.12
uvicorn                      0.22.0
validators                   0.20.0
watchdog                     2.2.0
watchfiles                   0.19.0
wcwidth                      0.2.6
weasyprint                   58.1
webcolors                    1.13
webencodings                 0.5.1
websocket-client             1.5.1
websockets                   11.0.3
Werkzeug                     2.2.2
wheel                        0.38.4
widgetsnbextension           4.0.7
wrapt                        1.14.1
WTForms                      3.0.1
xarray                       2023.5.0
xarray-einstats              0.5.1
xgboost                      1.7.6
XlsxWriter                   3.1.0
xlwings                      0.30.4
xmltodict                    0.13.0
xxhash                       3.3.0
yarl                         1.8.2
zipp                         3.11.0
zopfli                       0.2.2

[talhaanwarch]

are you using conda ?

[bbartling]

No not using Conda. Just ordinary or "vanilla" python.

> py -3.10
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

I most likely just installed the .exe executable from and added PATH during the installation:

• https://www.python.org/downloads/release/python-3106/

I think this one:
`

Windows installer (64-bit)	Windows	Recommended	8f46453e68ef38e5544a76d84df3994c	28916488	SIG

[Windows installer (64-bit)](https://www.python.org/ftp/python/3.10.6/python-3.10.6-amd64.exe) Windows Recommended 8f46453e68ef38e5544a76d84df3994c 28916488 [SIG](https://www.python.org/ftp/python/3.10.6/python-3.10.6-amd64.exe.asc)`

[andreapagliacci]

I have the same issue in Mac OS X, with python 3.10, ctransformers 0.2.2.4 (the last) using Conda

Traceback (most recent call last):
File "/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/main.py", line 17, in
llm = AutoModelForCausalLM.from_pretrained('/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/model/llama-2-7b-chat.ggmlv3.q2_K.bin', model_type='llama')
File "/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/hub.py", line 173, in from_pretrained
return LLM(
File "/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/llm.py", line 238, in init
self._lib = load_library(lib, cuda=config.gpu_layers > 0)
File "/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/llm.py", line 125, in load_library
lib = CDLL(path)
File "/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/ctypes/init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/lib/avx2/libctransformers.dylib, 6): no suitable image found. Did find:
/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/lib/avx2/libctransformers.dylib: cannot load 'libctransformers.dylib' (load command 0x80000034 is unknown)
/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/lib/avx2/libctransformers.dylib: cannot load 'libctransformers.dylib' (load command 0x80000034 is unknown)