Qdrant 中的相关性反馈

pip install qdrant-relevance-feedback

from qdrant_client import QdrantClient

client = QdrantClient(
    url=<QDRANT_URL>,
    api_key=<QDRANT_API_KEY>,
    cloud_inference=True # to be able to use free embedding inference on Qdrant Cloud, for convenience & to reduce latency
)

COLLECTION_NAME = "documentation"

from qdrant_relevance_feedback.retriever import QdrantRetriever

retriever = QdrantRetriever("sentence-transformers/all-minilm-l6-v2")

RETRIEVER_VECTOR_NAME = None # None if it's a default vector or your named vector handle in Qdrant's collection

PAYLOAD_KEY = "text"

pip install fastembed

from qdrant_relevance_feedback.feedback import FastembedFeedback

feedback = FastembedFeedback("mixedbread-ai/mxbai-embed-large-v1")

from qdrant_relevance_feedback import RelevanceFeedback

relevance_feedback = RelevanceFeedback(
        retriever=retriever,
        feedback=feedback,
        client=client,
        collection_name=COLLECTION_NAME,
        vector_name=RETRIEVER_VECTOR_NAME,
        payload_key=PAYLOAD_KEY
)

TRAIN_LIMIT = 25

formula_params = relevance_feedback.train(
    queries=train_queries,
    limit=TRAIN_LIMIT,
)

On 22.00% of training queries the feedback model strongly disagreed with the retriever model.

Naive formula params: a=0.240764, b=1.348897, c=0.590883

query = "recommendations API how to use"

query_embedding = retriever.embed_query(query)
CONTEXT_LIMIT = 3

responses = client.query_points(
    collection_name=COLLECTION_NAME,
    query=query_embedding,
    with_payload=True,
    with_vectors=True,
    limit=CONTEXT_LIMIT,
    using=RETRIEVER_VECTOR_NAME,
).points

responses_raw = [r.payload[PAYLOAD_KEY] for r in responses]

print("\n--- Initial Retrieval Results ---")
for i, text in enumerate(responses_raw, 1):
    print(f"  [{i}] {text[:200]}")
print()

--- Initial Retrieval Results ---
  [1] Recommendation API
  [2] recommendation API.
  [3] So, even when the API is not called recommend, recommendation systems can also use this approach and adapt it for their specific use-cases.

feedback_model_scores = feedback.score(query, responses_raw)

from qdrant_client import models

responses_point_ids = [p.id for p in responses] # CONTEXT_LIMIT responses from initial vanilla retrieval (their point IDs)
#responses_vectors = [p.vector for p in responses]

relevance_feedback_responses = client.query_points(
    collection_name=COLLECTION_NAME,
    query=models.RelevanceFeedbackQuery(
        relevance_feedback=models.RelevanceFeedbackInput(
            target=query_embedding,
            feedback=[
                models.FeedbackItem(example=example, score=score)
                for example, score in zip(responses_point_ids, feedback_model_scores) # or zip(responses_vectors, feedback_model_scores)
            ],
            strategy=models.NaiveFeedbackStrategy(
                naive=models.NaiveFeedbackStrategyParams(**formula_params) # our parameters from above
            ),
        )
    ),
    with_payload=True,
    limit=3,
    using=RETRIEVER_VECTOR_NAME,
).points

relevance_feedback_responses_raw = [r.payload[PAYLOAD_KEY] for r in relevance_feedback_responses]

print("\n--- Additional Results from Relevance Feedback-based Retrieval ---")
for i, text in enumerate(relevance_feedback_responses_raw, 1):
    print(f"  [{i}] {text[:200]}")
print()

--- Additional Results from Relevance Feedback-based Retrieval ---
  [1] Recap of the old recommendations API
  [2] The result of this API contains one array per recommendation requests.
  [3] Deliver Better Recommendations with Qdrant's new API

from qdrant_relevance_feedback.evaluate import Evaluator

N = 10  # as in metric@N

evaluator = Evaluator(relevance_feedback=relevance_feedback)

results = evaluator.evaluate_queries(
    at_n=N,
    formula_params=formula_params, # from above
    eval_queries=test_queries,
    eval_context_limit=CONTEXT_LIMIT # from above
)

On the 2nd retrieval iteration, over this test set:
  Relevance feedback retrieval surfaced 44 more relevant results (according to the feedback model)
  Vanilla retrieval surfaced 39 more relevant results (according to the feedback model)

Relative results relevance gain over this test set is: 12.82051282051282%

DCG win rates (which approach ranks results better):
  Vanilla retrieval:            38.0% wins
  Relevance feedback retrieval:  48.0% wins
  Ties:                          14.0%