How to use langchain load_evaluator() with local llm?

Question

I try to use langchain load_evaluator() with local LLM Ollama. But I don't understand which model I should use.

from langchain.evaluation import load_evaluator
from langchain.chat_models import ChatOllama
from langchain.llms import Ollama
from langchain.embeddings import HuggingFaceEmbeddings

#This is work
evaluator = load_evaluator("labeled_score_string", llm=ChatOllama(model="llama2"))
evaluator = load_evaluator("pairwise_string",  llm=Ollama(model="llama2"))

#This is not
evaluator = load_evaluator("pairwise_embedding_distance",  llm=HuggingFaceEmbeddings())
evaluator = load_evaluator("pairwise_embedding_distance",  llm=Ollama(model="llama2"))

Welcome to the SO community! I think your question is a bit broad to be well-received on the site. Can you add more details? What exactly is not working? Are there any error messages? Does it work with other LLMs rather than Ollama? — James
– James, Commented Mar 27, 2024 at 16:38

Dehan · Accepted Answer · 2024-11-29 12:33:36Z

0

I think we have same problem and I found this

embedding_function = OllamaEmbeddings(model="llama3.2:3b")
evaluator = load_evaluator("pairwise_embedding_distance", embeddings=embedding_function)

instead asking for "llm" parameter, it should provide "embeddings" param

source: https://python.langchain.ac.cn/docs/guides/productionization/evaluation/comparison/pairwise_embedding_distance/

answered Nov 29, 2024 at 12:33

Dehan

212 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to use langchain load_evaluator() with local llm?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related