quaterion.eval.pair.retrieval_reciprocal_rank module

class RetrievalReciprocalRank(distance_metric_name: ~quaterion.distances.Distance = Distance.COSINE, reduce_func: ~typing.Callable | None = <built-in method mean of type object>)

Bases: PairMetric

Calculates retrieval reciprocal rank for pair based datasets

Calculates the reciprocal of the rank at which the first relevant document was retrieved.

Parameters:

• distance_metric_name – name of a distance metric to calculate distance or similarity matrices. Available names could be found in Distance.

• reduce_func – function to reduce calculated metric. E.g. torch.mean, torch.max and others. functools.partial might be useful if you want to capture some custom arguments.

Example

Response on a query returned 10 documents, 3 of them are relevant. Assume positions of relevant documents are [2, 5, 9]. Then retrieval reciprocal rank being calculated as 1/2 = 0.5.

raw_compute(distance_matrix: Tensor, labels: Tensor)

Compute retrieval precision

Parameters:

• distance_matrix – matrix with distances between embeddings. Assumed that distance from embedding to itself is meaningless. (e.g. equal to max element of matrix + 1)

• labels – labels to compute metric. Assumed that label from object to itself has been made meaningless. (E.g. was set to 0)

Returns:

torch.Tensor - computed metric

retrieval_reciprocal_rank(distance_matrix: Tensor, labels: Tensor)

Calculates retrieval reciprocal rank given distance matrix and labels

Parameters:

• distance_matrix – distance matrix having max possible distance value on a diagonal

• labels – labels matrix having False or 0. on a diagonal

Returns:

torch.Tensor – retrieval reciprocal rank