Is there a way to automate formatting terminal input when talking to an LLM model?

Question

Currently when the user sends input to the LLM it has to be in the format of:

<|im_start|>user\
User input here.<|im_end|>

This is very cumbersome and makes prompt engineering very tedious. Right now i am using llama.cpp to run a llama2 model locally in the terminal - is there a way to automatically insert user input between the two tokens (im_start and im_end)?

I might have approached this problem in the wrong way, but the following is what I have tried:

My current (so far unsuccessful) method is to use a bash script to launch the LLM in another terminal and redirecting the output of the current terminal to the LLM-terminal. Later I was going to make a script that the first "launcher-terminal" would, which would format all user input and send it to the LLM-terminal.

Sofar I haven't been able to get the communication between the terminals working. The script would be trivial for me to make. It is moreso the bash-scripting that I'm struggling with.

I suspect my bash implementation is highly flawed, I'm not very experienced with it.

RUN_PATH="/llama.cpp/main"
MODEL_PATH="/llama.cpp/models/dolphin-2.9-llama3-8b-q4_K_M.gguf"
INIT_PATH="prompt.txt"

FLAGS="-n 512 --keep -1 --ctx-size 2048 --temp 0.0"

LAUNCH_CMD=".${RUN_PATH} -m .${MODEL_PATH} ${FLAGS} --color  --interactive-first --file .${INIT_PATH}"

# gnome-terminal -- "${LAUNCH_CMD}"
# Here im launching an impostor python program which just prints the input it receives.
gnome-terminal -- python3 impostor.py

LAUNCHER_PID="${$}"
LLM_PID="$PPID"

# It seems I don't get the PIDs i expect to get.
echo "Launcher PID: ${LAUNCHER_PID}"
echo "LLM PID:  ${LLM_PID}"

# Set STD-output of the launcher to LLM
ps -p $LLM_PID << ps -p $LAUNCHER_PID
/bin/bash

When typing echo hello world in the launcher terminal, it does not show up in the LLM-terminal, which is the desired effect.

Don't use all upper case for non-exported variable names, see correct-bash-and-shell-script-variable-capitalization. Also, regarding LAUNCH_CMD="..." - variables are for storing data, functions are for storing code, see mywiki.wooledge.org/BashFAQ/050. Finally, make sure to run your code through shellcheck.net and fix the other issues it'll tell you about. Once you've done that, if you still have a problem then edit your question to show the corrected script. — Ed Morton
– Ed Morton, Commented May 6, 2024 at 15:26

Diego Torres Milano · Accepted Answer · 2024-05-06 20:14:38Z

0

If you want to connect to the standard input and output of a running process you can use a coprocess.

Something like

coproc ./impostor.py

and then you can read queries, modify them, and send to the coprocess

while read -rp 'Prompt: ' p; do printf '<|im_start|>user\\\n%s<|im_end|>\n' "$p" >&"${COPROC[1]}"; done

To read the responses read from ${COPROC[0]}

answered May 6, 2024 at 20:14

Diego Torres Milano

69.9k9 gold badges116 silver badges145 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Is there a way to automate formatting terminal input when talking to an LLM model?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related