quaterion.eval.pair.retrieval_precision module

class RetrievalPrecision(k=1, distance_metric_name: ~quaterion.distances.Distance = Distance.COSINE, reduce_func: ~typing.Callable | None = <built-in method mean of type object>)

Bases: PairMetric

Calculates retrieval precision@k for pair based datasets

Parameters:

• k – number of documents among which to search a relevant one

• distance_metric_name – name of a distance metric to calculate distance or similarity matrices. Available names could be found in Distance.

• reduce_func – function to reduce calculated metric. E.g. torch.mean, torch.max and others. functools.partial might be useful if you want to capture some custom arguments.

Example

Assume k is 4. Then only 4 documents being retrieved as a query response. Only 2 of them are relevant and score will be 2/4 = 0.5.

Note

If k is greater than overall amount of relevant documents, then precision@k will always have score < 1.

raw_compute(distance_matrix: Tensor, labels: Tensor)

Compute retrieval precision

Parameters:

• distance_matrix – matrix with distances between embeddings. Assumed that distance from embedding to itself is meaningless. (e.g. equal to max element of matrix + 1)

• labels – labels to compute metric. Assumed that label from object to itself has been made meaningless. (E.g. was set to 0)

Returns:

torch.Tensor - computed metric

retrieval_precision(distance_matrix: Tensor, labels: Tensor, k: int)

Calculates retrieval precision@k given distance matrix, labels and k

Parameters:

• distance_matrix – distance matrix having max possible distance value on a diagonal

• labels – labels matrix having False or 0. on a diagonal

• k – number of documents to retrieve

Returns:

torch.Tensor – retrieval precision@k for each row in tensor