quaterion.eval.pair.retrieval_precision module¶
- class RetrievalPrecision(k=1, distance_metric_name: ~quaterion.distances.Distance = Distance.COSINE, reduce_func: ~typing.Optional[~typing.Callable] = <built-in method mean of type object>)[source]¶
Bases:
PairMetric
Calculates retrieval precision@k for pair based datasets
- Parameters:
k – number of documents among which to search a relevant one
distance_metric_name – name of a distance metric to calculate distance or similarity matrices. Available names could be found in
Distance
.reduce_func – function to reduce calculated metric. E.g. torch.mean, torch.max and others. functools.partial might be useful if you want to capture some custom arguments.
Example
Assume k is 4. Then only 4 documents being retrieved as a query response. Only 2 of them are relevant and score will be 2/4 = 0.5.
Note
If k is greater than overall amount of relevant documents, then precision@k will always have score < 1.
- raw_compute(distance_matrix: Tensor, labels: Tensor)[source]¶
Compute retrieval precision
- Parameters:
distance_matrix – matrix with distances between embeddings. Assumed that distance from embedding to itself is meaningless. (e.g. equal to max element of matrix + 1)
labels – labels to compute metric. Assumed that label from object to itself has been made meaningless. (E.g. was set to 0)
- Returns:
torch.Tensor - computed metric
- retrieval_precision(distance_matrix: Tensor, labels: Tensor, k: int)[source]¶
Calculates retrieval precision@k given distance matrix, labels and k
- Parameters:
distance_matrix – distance matrix having max possible distance value on a diagonal
labels – labels matrix having False or 0. on a diagonal
k – number of documents to retrieve
- Returns:
torch.Tensor – retrieval precision@k for each row in tensor