Pass raw prompt in Llama 3 prompting format ot LLama.cpp webserver

Question

I have an application running a local llama.cpp instance with llama 3. LLama 3 uses a new prompting format:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful AI assistant for travel tips and recommendations<|eot_id|><|start_header_id|>user<|end_header_id|>

What is France's capital?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Bonjour! The capital of France is Paris!<|eot_id|><|start_header_id|>user<|end_header_id|>

What can I do there?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Paris, the City of Light, offers a romantic getaway with must-see attractions like the Eiffel Tower and Louvre Museum, romantic experiences like river cruises and charming neighborhoods, and delicious food and drink options, with helpful tips for making the most of your trip.<|eot_id|><|start_header_id|>user<|end_header_id|>

Give me a detailed list of the attractions I should visit, and time it takes in each one, to plan my trip accordingly.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

So I need to generate these on the fly and pass them to llama. But llama.cpp seems not have any kind of option to pass the raw prompt, only a user prompt and a way to pass the system prompt (which is not enough I want a full partial conversation to be passed). So how can I do this with the llama.cpp web server?

Essentially how you can pass partial conversations as prompts with openai

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

user2741831 · Accepted Answer · 2024-05-29 13:24:28Z

0

It was my mistake. It does infact support raw queries per default, I just forgot that the default context size was 512 (although I don't know why that is). But increasing it fixed my problem

answered May 29, 2024 at 13:24

user2741831

2,4823 gold badges30 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pass raw prompt in Llama 3 prompting format ot LLama.cpp webserver

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related