quaterion.eval.group.retrieval_r_precision module¶
- class RetrievalRPrecision(distance_metric_name: Distance = Distance.COSINE)[source]¶
Bases:
GroupMetric
Compute the retrieval R-precision score for group based data
Retrieval R-Precision is the ratio of r/R, where R is the number of the relevant documents for a given query in the collection, and r is the number of the truly relevant documents found in the R highest scored results for that query.
- Parameters:
distance_metric_name – name of a distance metric to calculate distance or similarity matrices. Available names could be found in
Distance
.
Example
Suppose that a collection contains 20 relevant documents for our query, and the model can retrieve 15 of them in the 20 highest scored results, then Retrieval R-Precision is calculated as r/R = 15/20 = 0.75.
- raw_compute(distance_matrix: Tensor, labels: Tensor) Tensor [source]¶
Perform metric computation on ready distance_matrix and labels
This method does not make any data and labels preparation. It is assumed that distance_matrix has already been calculated, required changes such masking distance from an element to itself have already been applied and corresponding labels have been prepared.
- Parameters:
distance_matrix – distance matrix ready to metric computation
labels – labels ready to metric computation with the same shape as distance_matrix. Possible values are in {0, 1}.
- Returns:
torch.Tensor - calculated metric value
- retrieval_r_precision(distance_matrix: Tensor, labels: Tensor)[source]¶
Calculates retrieval r precision given distance matrix and labels
- Parameters:
distance_matrix – distance matrix having max possible distance value on a diagonal
labels – labels matrix having False or 0. on a diagonal
- Returns:
torch.Tensor – mean retrieval r precision