quaterion.eval.pair.retrieval_reciprocal_rank module¶
- class RetrievalReciprocalRank(distance_metric_name: ~quaterion.distances.Distance = Distance.COSINE, reduce_func: ~typing.Callable | None = <built-in method mean of type object>)[source]¶
Bases:
PairMetric
Calculates retrieval reciprocal rank for pair based datasets
Calculates the reciprocal of the rank at which the first relevant document was retrieved.
- Parameters:
distance_metric_name – name of a distance metric to calculate distance or similarity matrices. Available names could be found in
Distance
.reduce_func – function to reduce calculated metric. E.g. torch.mean, torch.max and others. functools.partial might be useful if you want to capture some custom arguments.
Example
Response on a query returned 10 documents, 3 of them are relevant. Assume positions of relevant documents are [2, 5, 9]. Then retrieval reciprocal rank being calculated as 1/2 = 0.5.
- raw_compute(distance_matrix: Tensor, labels: Tensor)[source]¶
Compute retrieval precision
- Parameters:
distance_matrix – matrix with distances between embeddings. Assumed that distance from embedding to itself is meaningless. (e.g. equal to max element of matrix + 1)
labels – labels to compute metric. Assumed that label from object to itself has been made meaningless. (E.g. was set to 0)
- Returns:
torch.Tensor - computed metric
- retrieval_reciprocal_rank(distance_matrix: Tensor, labels: Tensor)[source]¶
Calculates retrieval reciprocal rank given distance matrix and labels
- Parameters:
distance_matrix – distance matrix having max possible distance value on a diagonal
labels – labels matrix having False or 0. on a diagonal
- Returns:
torch.Tensor – retrieval reciprocal rank