How can I install llama-cpp-python with cuBLAS using poetry?

Question

I can install llama cpp with cuBLAS using pip as below:

CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python

However, I don't know how to install it with cuBLAS when using poetry. Installation is possible, but cuBLAS Acceleration is not available.
I checked that I can use cuBLAS when I installed it with pip in my environment.

I added llama-cpp-python dependency to the pyproject.toml file as below:

[tool.poetry.dependencies]
python = ">=3.10, <3.13"
...
llama-cpp-python = "^0.2.13"
...

I tried

CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 poetry install

And

export CMAKE_ARGS="-DLLAMA_CUBLAS=on"
export FORCE_CMAKE=1
poetry install

Adrian Mole · Accepted Answer · 2023-12-15 17:20:46Z

6

I encountered a similar issue and found a workaround. While Poetry doesn't directly support passing environment variables like pip, I used poetry run pip install as a temporary solution. This approach involves setting the necessary environment variables and then running:

poetry run pip install llama-cpp-python --upgrade --force-reinstall --no-cache-dir

This method allowed me to install llama-cpp-python with CU-BLAS support, which I couldn't achieve solely with Poetry. It's important to note that this bypasses Poetry's dependency resolution, so use it cautiously and document it in your project.

I tried using https://github.com/volopivoshenko/poetry-plugin-dotenv/, but was still not getting it to work.

edited Dec 15, 2023 at 17:20

Adrian Mole

52.2k193 gold badges61 silver badges101 bronze badges

answered Dec 7, 2023 at 20:33

Luis Antunes

611 bronze badge

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How can I install llama-cpp-python with cuBLAS using poetry?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related