How to compose Ollama server with Nextjs?

Question

I have been attempting to create a composition involving the Ollama server with LLMs such as Mistral and Llama2, along with a Next.js server to interact with it. I am building this project to learn Docker, and I am currently facing an issue.

So I have created a docker file for Nextjs like this :

FROM node:18-alpine

WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install

COPY . .
RUN npm run build
CMD ["npm", "start"]

Along with this I seek to run a llama server with cached dependencies. For this I tried to create a docker-compose file :

version: '3.8'
services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    command: ollama pull llama2

  ollama-ui:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "3000:3000"

This way I am trying to run a command called ollama pull llama2, intending for llama2 to be pulled before it starts Nextjs server. But there is an error :

ollama-1 | Error: unknown command "ollama" for "ollama".

Goal of this project is a composed container running ollama server with some LLMs running along with my Nextjs server and I want the process of pulling LLMs to be cached. Is this a proper approach? If not what should it be?

I am sure this is some silly problem but will be glad if someone schools me here.

Thanks!

Sebastian Boutin Blomfield · Accepted Answer · 2024-02-07 19:31:36Z

3

In case anyone is still looking for a better solution, the issue is that the docker image's entrypoint is already the ollama command, so you can just directly do pull llama2 without the ollama bit.

EDIT: Unfortunately this causes a different issue, because docker-compose doesn't easily let you start the server and then run the pull command, so the API approach is still the best for actually achieving the desired goal.

edited Feb 7, 2024 at 19:31

answered Feb 7, 2024 at 17:33

Sebastian Boutin Blomfield

413 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

rusty_python · Accepted Answer · 2024-01-26 04:38:54Z

2

I have faced the same problem and found that despite being in /bin, compose can't find the ollama command while running the image. One hack I've found around this is to run a script after docker compose is running to pull llama2.

#!/usr/bin/bash

curl -X POST \
-H "Content-Type: application/json" \
-d '{"name":"llama2"}' \
 http://localhost:11434/api/pull

answered Jan 26, 2024 at 4:38

rusty_python

1269 bronze badges

3 Comments

Mohit Ranjan Over a year ago

Thanks a lot, I ended up checking for existing models and pulling them via API itself.

Aishwarya Patange Over a year ago

I had a doubt. Sorry if it sounds dumb. Where do you exactly run this script? Is it in the app Dockerfile? like RUN /path/to/script.sh in the ollama-ui Dockerfile?

rusty_python Over a year ago

@AishwaryaPatange You can run it from your shell as the ollama service gets exposed to your host machine via port 11434.

Collectives™ on Stack Overflow

How to compose Ollama server with Nextjs?

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related